Evaluation design
Overview
The design phase is where groups make specific decisions about what information will be collected, how, and when. In an ideal situation, these decisions would be driven by the questions the group has chosen to ask, and the level of specificity, surety and timeliness needed to answer the questions. In the real world, evaluation design usually considers those issues, but final decisions are often made based on available capacities and resources for evaluation, the kinds of information that key audiences believe matters, the difficulty or ease of getting data from various people, groups and systems, and other real world issues. This is a key step to watch for privilege influencing the work. Privilege infects evaluation when these real world issues override the integrity of evaluation – often not with that intent, but with that impact. This happens when, for example, communities or programs are judged as “failures” based on looking for results too soon, but reporting them anyway, or failing to control for inequitable distribution of opportunities and system resources in looking at why certain groups score less well on health, education and income measures.
The challenge then is to figure out the bottom-line issues that affect the integrity of evaluation, on which a group cannot compromise. For example, if a group knows that it will take at least three years for its work to begin to reduce inequitable health outcomes in a particular health system, the interim outcomes, questions asked, timing of data collection and ways in which findings are reported have to align with that understanding. In practical terms, this means setting a context and realistic expectations as the evaluation is being designed, and keeping that context and those expectations front and center as data are reported. It might also mean being sure to design differently (in terms of methods and questions) for different stages of work (for example, start-up, refinement, early results, expansion or retrenchment, replication).
Tipsheets
- How Confident Do We Have To Be In Our Evaluation Results? What Should We Consider In Terms Of Evaluation Resources Vs. Rigor?
- How Can The Effects Or Impacts Of Our Strategies Be Measured?
Resources
- Evaluating Comprehensive Community Change
- Getting Smart, Getting Real: Using Research and Evaluation Information to Improve Programs and Policies
- We Did it Ourselves: An Evaluation Guidebook (Chapter 11)
- Ways to Improve the Quality of Your Program Evaluations
- EvaluLead - A Guide for Shaping and Evaluating Leadership Development Programs
- Evidence-based Practices, Practices-based Evidence and Community defined Evidence in Multicultural Mental Health
- Greater Portland Pulse: Challenges to Measuring Racial Equity
- Evaluating Complexity: Propositions for Improving Practice
- Raising the Bar – Integrating Cultural Competence and Equity: Equitable Evaluation
- Medicine Wheel Evaluation Framework
- Evaluating Social Innovation - the What and the When
- Conducting Culturally Competent Evaluations of Child Welfare Programs and Practices
- Evaluation of Capacity Building: Lessons from the Field
- Considerations for Conducting Evaluation Using a Culturally Responsive and Racial Equity Lens
- How to Design and Manage Equity-focused Evaluations
- LGBT-Inclusive Language in Data Collection
- Universal Design Evaluation for Checklist
- Historical Trauma and Unresolved Grief: Implications for Clinical Research and Practice with Indigenous Peoples of the Americas
- How Do We Approach Impact and Evaluation in the Context of Scale?
- Considerations for Conducting Evaluation Using a Culturally Responsive and Racial Equity Lens
- Collaborating to See All Constituents Reach Their Full Potential: Memorandum on Research and Resources on Equity and Collective Impact
- Moving Toward Equity Stakeholder Engagement Guide
- Discussion Guide: Theory of Social Change
Tools
- W.K. Kellogg Foundation Evaluation Handbook
- W.K. Kellogg Foundation Logic Model Development Guide
- Avoid These Mistakes in Your Program Evaluation
- The Advocacy Progress Planner
- Community Development Evaluation Storymap and Legend
- Processes and Indicators for Measuring the Impact of Equality Bodies
- Advocacy Evaluation Mini-Toolkit: Tips and Tools for Busy Organizations
- Equal Access Participatory Monitoring and Evaluation Toolkit
- Considering Culture: Building the Best Evidence-based Practices for Children of Color
