Designing impact evaluations: different perspectives - 3ie Working Paper 4

The starting point for an evaluation is to ask why it is being conducted, who will benefit, and what impact the evaluation will itself have, and how. Participatory approaches and methods fit in a paradigm that is pluralist, evolutionary and iterative. They include stakeholder analysis, individual story-telling, participatory social mapping, causal-linkage and trend and change diagramming, scoring, and brainstorming on program strengths and weaknesses. Well designed and facilitated, participatory methods are rigorous, and besides offering qualitative insights can count the uncountable, and generate statistics for relevant dimensions that would otherwise be overlooked or regarded as purely qualitative. They open studies to the voices of those most affected by a project in a ways not possible using more conventional methods and can make the realities and experiences of poor people count more.

In all three cases considered here – a CCT, infrastructure development, and an anti- corruption commission - I would argue for an approach that was pluralist, evolutionary and iterative. Mixed methods would be used. The starting point would be to ask about the political economy of the evaluation: who would gain? Who might lose? And how? And, especially, how was it intended and anticipated that the findings would make a difference. This might well require a brainstorming workshop with staff from the funding agency. If they were unwilling or unable to find the time for this, or to shed light on these questions, I hope I would have the guts and resources to turn down the assignment. I would negotiate the MOU to include other steps. One would be a stakeholder analysis and negotiations to involve relevant stakeholders in the process.