Evaluation design


The design phase is where groups make specific decisions about what information will be collected, how, and when. In an ideal situation, these decisions would be driven by the questions the group has chosen to ask, and the level of specificity, surety and timeliness needed to answer the questions. In the real world, evaluation design usually considers those issues, but final decisions are often made based on available capacities and resources for evaluation, the kinds of information that key audiences believe matters, the difficulty or ease of getting data from various people, groups and systems, and other real world issues. This is a key step to watch for privilege influencing the work. Privilege infects evaluation when these real world issues override the integrity of evaluation – often not with that intent, but with that impact. This happens when, for example, communities or programs are judged as “failures” based on looking for results too soon, but reporting them anyway, or failing to control for inequitable distribution of opportunities and system resources in looking at why certain groups score less well on health, education and income measures.

The challenge then is to figure out the bottom-line issues that affect the integrity of evaluation, on which a group cannot compromise. For example, if a group knows that it will take at least three years for its work to begin to reduce inequitable health outcomes in a particular health system, the interim outcomes, questions asked, timing of data collection and ways in which findings are reported have to align with that understanding. In practical terms, this means setting a context and realistic expectations as the evaluation is being designed, and keeping that context and those expectations front and center as data are reported. It might also mean being sure to design differently (in terms of methods and questions) for different stages of work (for example, start-up, refinement, early results, expansion or retrenchment, replication).




Also in this section: