Photo by Jason Goodman on Unsplash
STEP 1: CLARIFY WHAT IS TO BE EVALUATED
Purpose: to develop an evaluability assessment with clearly-defined goal(s), populations, strategies, activities, outputs, and outcomes.
Step 1 addresses whether an evaluation should be carried out given available information and current context. Begin with a clear description of what is to be evaluated (e.g. program, policy, or health communication initiative) so that it can be shared. Consider developing a logic model that includes goals, activities, outputs, and outcome objectives.
STEP 2: ENGAGE STAKEHOLDERS
Purpose: to build support for the evaluation by engaging stakeholders
This step involves identifying key stakeholders, understanding their interests and expectations, and engaging them in a review of objectives to develop evaluation questions. Consider making a list of organizations and/or people with an interest in the evaluation. Define stakeholder information needs and intended use of the evaluation.
STEP 3: ASSESS RESOURCES AND EVALUABILITY
Purpose: to assess available resources for evaluation and whether the program is ready to be evaluated.
Evaluation can be time-consuming and expensive. Do an honest assessment of resources to avoid constraints later. Resources include funds, time, in-kind support, approval processes (e.g., ethics) and timeline for implementation and completion. When assessing evaluability consider whether:
- there is clarity on the program to be evaluated and why you are doing an evaluation
- the evaluation will be useful
- leadership buy-in is high and will shape programs
- adequate resources are available
- timing is right
First, use this information to make a decision about whether conducting the evaluation is feasible and necessary. Then, use the information to further define your evaluation.
STEP 4: DETERMINE YOUR EVALUATION QUESTIONS
Purpose: to help identify and determine questions to meet your evaluation goals.
Involve as many stakeholders as possible to ensure all needs are met. Consider: logic model contents, stage of development (e.g., planning, implementation, or completion stage), evaluations already completed, decisions needed, stakeholder interests, and resources. Consider type of evaluation needed. Formative evaluation is typically used in development and planning of a strategy. Process evaluation addresses the extent to which an intervention is implemented as planned and reaching intended populations. Outcome evaluation measures a program’s success in meeting its goals and objectives.
STEP 5: DETERMINE APPROPRIATE METHODS OF MEASUREMENT AND PROCEDURES
Purpose: to identify the most feasible and credible methods to use and how data will be collected.
Consider: purpose, evaluation questions, feasibility of data access, realiability of data, who will use data, stakeholder expectations, and what is already being captured. Develop a data collection plan that includes: what to measure (indicators), when to collect data (before/after/both), how to collect data (qualitative/quantitative/both) and from whom to collect data (specific sub-groups/representative sample of population of interest). Think about the ethical issues involved in data collection (e.g. anonymity of data, confidentiality, and informed consent) and take appropriate steps.
STEP 6: DEVELOP EVALUATION PLAN
Purpose: to identify specific evaluation activities, tasks, roles, resource allocations, and deadlines.
Evaluation plans detail how programs, policies, or health communication initiatives will be monitored and evaluated, and how results will be used, providing transparency to stakeholders and funders. A data collection matrix can be part of your evaluation plan. A matrix generally includes information gathered from previous steps: evaluation questions and link to logic model (if applicable), indicators, methods, data sources, timelines, roles and responsibilities, and how data will be analyzed (optional). In Step 9 you will develop your dissemination plan, which can be added to the evaluation plan once completed.
STEP 7: COLLECT DATA
Purpose: to collect credible evidence to answer each evaluation question - results and recommendations depend upon data quality.
Develop data collection tools (survey, interview guide, etc.) and procedures and train data collectors. Consider whether incentives are appropriate and brainstorm ways to enhance response rates. To ensure validity, pilot test tools and procedures and closely monitor data gathered. If issues arise, modify tools and procedures and document changes. Computerize data collection to facilitate analysis if appropriate.
STEP 8: PROCESS DATA AND ANALYZE RESULTS
Purpose: to enter data, check quality and consistency of data entry and analyze data to identify your evaluation results. Implement strategies to review data quality during and after data collection. During data collection, look closely at the first wave of responses and number of ‘no response’ or refusals, and keep data collectors and the evaluation lead connected. After data collection, enter data and double check quality and consistency of entry, sort to find missing, high or low values (quantitative), and check content by reviewing transcripts entered (qualitative). Organize data in a format that can be summarized and interpreted. Analyze by conducting statistical analysis of quantitative data; identify themes in qualitative data. This is a technical step - enlist expert support when possible. This sets the stage for interpretation.
STEP 9: INTERPRET AND DISSEMINATE RESULTS
Purpose: to interpret and share your evaluation findings, engaging stakeholders so that they can help identify recommendations.
Anchor the interpretation to the original evaluation questions. Create a list of recommended actions that address your outcomes, and use this information to create the materials to communicate your findings. Presentation of findings can take many forms such as a written report, slide show presentation, and/or as a short informational video. Visual aids can be powerful methods for communicating evaluation results. Make results available to various stakeholders and audiences. Tailor what is disseminated to their specific interest in the evaluation and how they plan to use the results.
STEP 10: APPLY EVALUATION FINDINGS
Purpose: to use your evaluation results.
Review recommendations with stakeholders to identify actionable outcomes and discuss what has been learned from conducting the evaluation and next steps to incorporate results. Prioritize actions and develop an action plan as a group. Consider evaluating your evaluation (meta-evaluation). You may ask stakeholders to reflect on the process and/or the outcomes, to improve the process moving forward.
Source: Contributors—Susan Snelling, Allison Meserve, and Kim Bergeron, Health Promotion Consultant, Health Promotion Capacity Building, Public Health Ontario
Citation: Ontario Agency for Health Protection and Promotion
