ABSTRACT EVALUATION CAPACITY BUILDING (ECB) - VIRTUALLY
Lawmakers and the public have pushed for increased accountability in health and social programs. As a result,
evaluators and evaluation skills have become very important to both large and small organizations providing
health and social service programs. However, much of the public health workforce is not well equipped to
handle these increased demands for outcomes data and research. Many federal agencies (e.g., Health and
Human Services, Department Labor, Department of Justice) and private foundations are beginning to fund only
organizations that deploy evidence-based interventions demonstrating impact on behaviors (e.g., individual,
organizational, community). Our firm, through its national evaluation work has witnessed, first hand, the
challenges faced by the staff and leadership of many organizations engaged in planning and responding to
health issues, including (1) lack of expertise in evaluation and the ability to capture data related to program
outcomes; (2) adverse experiences with external evaluators and researchers; (3) lack of resources, limited
time and inadequate funds for independent evaluations; and, (4) fear of negative evaluation results. These
inhibitors are also cited in literature and reports on evaluation capacity building. Evaluation can require
considerable time and resources and many organizations cannot afford to create separate evaluation
departments or hire external evaluators.
EVALUATION CAPACITY BUILDING - VIRTUALLY (ECB-V) is a product concept that has emerged from our firm’s 23
years of work in program planning and evaluation with government and nongovernment organizations. We
envision ECB-V as customized, multimedia Elearning course (ECourse) with a supplemental EAssistant to
transfer learning from the individual to the program and organizational levels. The goal of this multi-level
Elearning and engagement tool is to improve the capacity of the program staff of community-based
organizations and state and local health departments to plan, implement and evaluate programs that advance
population health in the U.S. The REESSI staff will focus on two key aims in carrying out this project – (1) Our
team will develop the ECB-V prototype through iterative activities with experts and end-users and (2) evaluate
the ECB-V prototype using quantitative and qualitative methods.
The transdisciplinary team of population and public health experts, evaluators and software developers will
engage in this proposed project to develop the prototype and conduct a feasibility study to determine the
degree to which program staff and independent evaluators find the ECourse and EAssistant appealing, usable
and valuable (feasibility) and how the ECourse and EAssistant affect program staff knowledge of program
planning and evaluation (merit). The development of the user process for the product will be guided by the
Evaluation Capacity Building Theory of Change Model to design and deliver stimulating and interactive content
with electronic surveys in a digital environment. ECB-V will consist of a comprehensive ECourse with eight
learning modules and an EAssistant for on-the-job tasks. The proposed topics (content) for the ECourse are
based on (1) the Centers for Disease Control and Prevention Evaluation Guide and (2) the Office of Minority
Health Evaluation Protocol.
The Phase I activities will provide the foundation for the construction of the full and final components of the
ECourse and EAssistant. The Phase II activities will consist of a multi-site randomized controlled evaluation
that focuses on both individual and organization outcomes (e.g., attitudes, evaluation knowledge, public health
core competency skills, reported changes in organizational systems and decision making).
