Varna’s Mind

By Sherine Jayawickrama

A recent Harvard Business School working paper titled The Limits of Nonprofit Impact: A Contingency Framework for Measuring Social Performance (by Alnoor Ebrahim and V. Kasturi Rangan) comes to an interesting conclusion: that measuring impact only makes sense under a limited set of circumstances.

In their May 2010 paper, Ebrahim and Rangan consider the debates on performance and impact of nonprofit organizations playing out in private foundations, U.S. nonprofits and international development actors (both donor agencies and NGOs).  They observe that there has been increasing pressure on NGOs to measure performance at all levels of the logic chain: from inputs, activities and outputs (that are under their control) to broader outcomes and impacts (that they have little control over). 

Their review of evaluation literature finds that the term “impact evaluation” often refers to evaluations involving a counterfactual (what would have happened without the intervention being evaluated) and that experimental designs using randomized control trials (RCTs) are now considered the “gold standard” for assessing impact. 

Ebrahim and Rangan also acknowledge the literature challenging the notion of one “gold standard.”  RCTs are often unsuitable for evaluating work that seeks to influence complex development pathways which involve multiple, interacting and non-linear causal factors. The authors recognize the variety of new approaches to evaluation that try to combine methodological rigor with the ability to adapt to complex, non-linear systems.

Against this backdrop, Ebrahim and Rangan look at the variety of work that NGOs engage in (emergency and relief work, service delivery work of small vs large scale/scope, and advocacy and rights-based work) and explore how far down the logic chain results can reasonably be measured. They find that, except in the case of service delivery at significant scale and scope, it is not feasible for NGOs to measure performance beyond outputs and outcomes.  This presents a dilemma, especially for funders who increasingly seek evidence of impact.  

In response, Ebrahim and Rangan offer a contingency framework based on the notion that “what you should measure is contingent on what you’re trying to achieve.”  Their framework is built around two dimensions: an intervention’s theory of change (the causal logic underlying the intervention) and its operational strategy (how an organization implements interventions that advance its mission).  They use this framework to define four types of results, along with appropriate measures.

1. Niche Results - these results can be expected when theory of change and operational strategy are both focused (e.g. soup kitchens) and appropriate measures would be inputs, activities and outputs.
2. Integrated Results - these results can be expected when theory of change is focused but operational strategy is complex (i.e. humanitarian relief, health services) and appropriate measures would be aggregate outputs, outcomes and sometimes impacts.
3. Institutional Results - these results can be expected when theory of change is complex but operational strategy is focused (e.g. policy advocacy) and appropriate measures would be outputs and “influence” (or intermediate outcomes).
4. Ecosystem Results - these results can be expected when theory of change and operational strategy are both complex (e.g. rights-based approaches to development) and appropriate measures would be outcomes and impacts (over a timeframe in which systemic changes could reasonably have taken place).

Of course, the work of NGOs does not fall neatly into one of these four quadrants and the authors acknowledge that. Their intent is to move beyond the rather polarized debate on impact measurement - between those who want impact to be measured quantifiably (and equate that to rigor and accountability) and those who argue that development work is so context-specific that it is very hard to measure - to a conversation about where on the logic chain measurement should focus.

I found the paper interesting in that it challenges the notion that NGOs must demonstrate impact in order to prove that they are effective and accountable. The authors are realistic about how little control NGOs and their funders have over factors that either perpetuate social problems or contribute to their resolution; at the same time, they don’t let anyone off the hook.  Ebrahim and Rangan urge NGOs to measure results as far down the logic chain as reasonable, but they warn against going too far. That would lead to wasted resources, diverted attention and loss of credibility. 

The paper spurred some questions for me.  First, the most innovative NGO efforts cut across two or more quadrants that Ebrahim and Rangan identify (combining service delivery with capacity building, empowerment, policy advocacy, etc.). To the extent that this work exemplifies complex theories of change and operational strategies, the contingency framework warrants measurement of outcomes and impact. Yet, these efforts often work within messy, non-linear systems over which NGOs have little control.  While an NGO’s contribution to progress can be evaluated, the attribution of results is problematic.  What types of measures (and methods) would be most valuable for learning, most credible for upward accountability and most meaningful for downward accountability?

Second, having a focused theory of change in a complex setting (e.g. emergency response operations in the wake of a natural disaster or in the midst of a conflict) may be necessary to drive implementation forward efficiently, but it might paper over social and political dynamics with which NGO interventions inevitably interact.  Does the contingency framework adequately address the importance of assessing unintended outcomes?

Finally, although there is much I like about this framework and its implications, it feels a bit distant from the main protagonists of development and social change: members of poor communities, their aspirations, their agency and their rights.  Management terminology and development jargon often have that effect!  How would this framework be different if NGO accountability were looking toward communities rather than (or in addition to) funders? 

Sherine Jayawickrama manages the Humanitarian & Development NGOs domain of practice - and the Humanitarian & Development NGOs blog - at the Hauser Center for Nonprofit Organizations at Harvard University.

I am sad about this news, I think this is an error by google to stop wave.

I think they did not give it an opportunity at the level of the revolution it brings.

sure the people were not widely welcoming but because they were not seeing things that could impact them and usable in a simple maner. Yes it is complex for people but it could be overcome, like Steve overcame the BS complexity of smartphones produced by MS.

but instead of stopping it they should have fostered the development of applications professional ones and other user friendly ones for every day people.

I still have an idea to use wave professionnally but I don’t know where to express it.

so I will do it here :

The idea is sharing trouble shooting of network incidents among technical people and for others less technical people, I think it could be used by Cisco (or their competitors) to provide a service for their customer.

context :

when there is a major network incident generally people at cisco set up a call conference open to technical people and above all to managers to exchange on the situation and ideas for decisions (changing hardware, rebooting etc.).

Meanwhile there is a Network Engineer from Cisco or from the service provider connecting to the network, one equipment after another to find the reason for the incident and trying to settle it for a while or definitely.

several poeple can log on different routers by it is not easy for them to exchange between each other and see what the others have found.

At the same time may be a chat between network engineer take place to exchange by text

idea :

The idea would be to have a bot in a wave that would copy all that is entered by the network engineer and replied by the router on a copy of the terminal screen in a wave (or blip or wavelet) so that several technical people (in the same wave) could share what other are doing or observing on their terminal.

an image of what is going on, on the terminal screen of one, or up to several technical personn’screen could be displayed in the wave and shared to other people (SP managers, Cisco technical people)

a chat could be shared among technical people

and may be even the call conf could be hosted in a wave.

to share information about what is going on in the trouble shooting session and exchange by audio at the same time as seeing what is going on the technical troubleshooters’screens.

wave could do that great, it could help a lot !

appart from the terminal copy (it could be a bot or an app to developp), every thing is already there in wave(chat, audio conf etc.), just needing the integrating glue, it could bring a lot to the team work of trouble shooting.

I have had this idea for a while but who to express it to ?

to cisco here in France (where I live) but most of the tools come from the USA ?

should I have expressed it to google ?

May be that the difficult part of google wave, it is a collaboration tool and people have to collaborate to make the most of its potential, “it takes two to tango”, and the two parts that would benefit may be, are not ready to collaborate at the same time ? one is willing and the other is not ready yet ?

a bit like Telephony over IP the requested people from telepony to work with people from IT ?

I think wave is revolutionnary, I hope it will be used (one day soon) at the level of what it can ultimatly bring to people !