Varna’s Mind

By Sherine Jayawickrama

A recent Harvard Business School working paper titled The Limits of Nonprofit Impact: A Contingency Framework for Measuring Social Performance (by Alnoor Ebrahim and V. Kasturi Rangan) comes to an interesting conclusion: that measuring impact only makes sense under a limited set of circumstances.

In their May 2010 paper, Ebrahim and Rangan consider the debates on performance and impact of nonprofit organizations playing out in private foundations, U.S. nonprofits and international development actors (both donor agencies and NGOs).  They observe that there has been increasing pressure on NGOs to measure performance at all levels of the logic chain: from inputs, activities and outputs (that are under their control) to broader outcomes and impacts (that they have little control over). 

Their review of evaluation literature finds that the term “impact evaluation” often refers to evaluations involving a counterfactual (what would have happened without the intervention being evaluated) and that experimental designs using randomized control trials (RCTs) are now considered the “gold standard” for assessing impact. 

Ebrahim and Rangan also acknowledge the literature challenging the notion of one “gold standard.”  RCTs are often unsuitable for evaluating work that seeks to influence complex development pathways which involve multiple, interacting and non-linear causal factors. The authors recognize the variety of new approaches to evaluation that try to combine methodological rigor with the ability to adapt to complex, non-linear systems.

Against this backdrop, Ebrahim and Rangan look at the variety of work that NGOs engage in (emergency and relief work, service delivery work of small vs large scale/scope, and advocacy and rights-based work) and explore how far down the logic chain results can reasonably be measured. They find that, except in the case of service delivery at significant scale and scope, it is not feasible for NGOs to measure performance beyond outputs and outcomes.  This presents a dilemma, especially for funders who increasingly seek evidence of impact.  

In response, Ebrahim and Rangan offer a contingency framework based on the notion that “what you should measure is contingent on what you’re trying to achieve.”  Their framework is built around two dimensions: an intervention’s theory of change (the causal logic underlying the intervention) and its operational strategy (how an organization implements interventions that advance its mission).  They use this framework to define four types of results, along with appropriate measures.

1. Niche Results - these results can be expected when theory of change and operational strategy are both focused (e.g. soup kitchens) and appropriate measures would be inputs, activities and outputs.
2. Integrated Results - these results can be expected when theory of change is focused but operational strategy is complex (i.e. humanitarian relief, health services) and appropriate measures would be aggregate outputs, outcomes and sometimes impacts.
3. Institutional Results - these results can be expected when theory of change is complex but operational strategy is focused (e.g. policy advocacy) and appropriate measures would be outputs and “influence” (or intermediate outcomes).
4. Ecosystem Results - these results can be expected when theory of change and operational strategy are both complex (e.g. rights-based approaches to development) and appropriate measures would be outcomes and impacts (over a timeframe in which systemic changes could reasonably have taken place).

Of course, the work of NGOs does not fall neatly into one of these four quadrants and the authors acknowledge that. Their intent is to move beyond the rather polarized debate on impact measurement - between those who want impact to be measured quantifiably (and equate that to rigor and accountability) and those who argue that development work is so context-specific that it is very hard to measure - to a conversation about where on the logic chain measurement should focus.

I found the paper interesting in that it challenges the notion that NGOs must demonstrate impact in order to prove that they are effective and accountable. The authors are realistic about how little control NGOs and their funders have over factors that either perpetuate social problems or contribute to their resolution; at the same time, they don’t let anyone off the hook.  Ebrahim and Rangan urge NGOs to measure results as far down the logic chain as reasonable, but they warn against going too far. That would lead to wasted resources, diverted attention and loss of credibility. 

The paper spurred some questions for me.  First, the most innovative NGO efforts cut across two or more quadrants that Ebrahim and Rangan identify (combining service delivery with capacity building, empowerment, policy advocacy, etc.). To the extent that this work exemplifies complex theories of change and operational strategies, the contingency framework warrants measurement of outcomes and impact. Yet, these efforts often work within messy, non-linear systems over which NGOs have little control.  While an NGO’s contribution to progress can be evaluated, the attribution of results is problematic.  What types of measures (and methods) would be most valuable for learning, most credible for upward accountability and most meaningful for downward accountability?

Second, having a focused theory of change in a complex setting (e.g. emergency response operations in the wake of a natural disaster or in the midst of a conflict) may be necessary to drive implementation forward efficiently, but it might paper over social and political dynamics with which NGO interventions inevitably interact.  Does the contingency framework adequately address the importance of assessing unintended outcomes?

Finally, although there is much I like about this framework and its implications, it feels a bit distant from the main protagonists of development and social change: members of poor communities, their aspirations, their agency and their rights.  Management terminology and development jargon often have that effect!  How would this framework be different if NGO accountability were looking toward communities rather than (or in addition to) funders? 

Sherine Jayawickrama manages the Humanitarian & Development NGOs domain of practice - and the Humanitarian & Development NGOs blog - at the Hauser Center for Nonprofit Organizations at Harvard University.