One country, seven districts, seven international NGOs, five million dollars (excluding food commodity monetization), and…

161 indicators.

How do you begin to collect, analyze and, more importantly, utilize data on 161 indicators in this scenario?

Got me. And I was the Technical Advisor on Monitoring & Evaluation for this project – ha!

What a beautiful, i.e. funded, proposal it was—concocted by the lead INGOs’ most well-respected, imported-from-headquarters PhDs, in absentia of any real meaningful consultation with implementing partners, i.e. local organizations who do the actual work with communities, let alone the people they intended to serve.

I've been working in monitoring & evaluation (M&E) for over ten years now and during this time in the international development sector as a whole, I've seen an increasing desperation to find “evidence” of what is often inherently beyond logic and induction, also discussed recently by Rick Davies, Ramaswami Balasubramaniam, Lawrence Haddad, Dennis Whittle, Ben Ramalingam, and Alanna Shaikh, just to name a few. Caroline Preston also reports on “the data dash” of recent years in the philanthropic sector, which too often overlooks how “personal relationships, social networks, family and community dynamics, passion for causes, and other factors” that shape change.
The delusion of thinking you can conquer your world leads you to lose your soul. ~Cornel West
I see the development and philanthropic sectors locked in an increasing fixation on solving the problem of poverty through reductive ways of measurement. However, abstract metrics and experimental design is quite far from the intimate, difficult, and complex factors at play at the community level. Thus I believe it is time to examine our belief that there are technocratic, precise ways of measuring progress in order to make consequential judgments based on these measures. The business sector seems to have a healthier relationship with risk, perhaps something we need to further explore in the development sector.

M&E implemented solely for the purpose of accountability time and again fails to result in improved programming and, in many cases, undermines the effectiveness of the very interventions it is trying to measure. (See related research paper by Blackett-Dibinga & Sussman.) And the latest trend towards using the "gold standard" of randomized control trials is especially troubling when one is talking about community-level initiatives. Imposing such incredibly risk-averse behavior through evaluating every single intervention can most certainly be a drain on the time and scarce resources of people who are in the process of organizing at the local level, let alone the development professionals engaged in supporting them.

As someone who has worked extensively in building the M&E capacity of grassroots organizations in Africa, what I have found is that abstract metrics or research frameworks don’t often help people understand their relationship to improving the well-being of those they serve. Rather than using any theory or logframes, local leaders, as members of a community, read real-time trends via observation of what’s happening on the ground, which, in turn, drives intuition. Most Significant Change, Outcome Mapping, and The Good Enough Guide are examples of alternative approaches to M&E that are better grounded in this reality.

For the past few years, I led the development of an innovative training and mentoring approach to monitoring and evaluation among local indigenous organizations in Lesotho, Zambia and Malawi. The approach is based on a premise that people at the grassroots level have the most expertise in terms of defining and measuring success, based on internal reflection processes of their own values and goals. One mantra at its core—M&E should never detract from the work at hand, which is serving people—kept it grounded in the practicalities of day-to-day work with families and communities. The approach’s success has also come from its focus on making M&E accessible through the “de-technicalization” of M&E concepts and practical exercises that utilized existing data from the organizations themselves, further developing critical reflection skills. (You can see an overview presentation of this training approach here.) Trained groups continue to demonstrate the effectiveness of the approach through their program adaptation, enhanced advocacy, and increased fundraising.
Those who work selfishly for results are miserable. ~Sri Krishna, Bhagavad Gita, The Song of God
Make no mistake. I’m not arguing that more rigorous evaluation techniques do not have their place, especially for larger, publicly-funded projects. I myself am a big data geek. A database full of numbers or a pile of raw stories to play with and I’m in my bliss, identifying trends and constructing potential causal inferences.

But as practitioners, we must always consider what is the appropriate cost and complexity needed for evaluation, especially given the size, scope, level of intervention, and length of the program. We must also aim for proportional expectations so we ensure M&E is a tool for learning and improvement, not just policing. Yet again, how matters.

Yes, let's pursue and obtain useful data from the ground, but at a scale at which information can be easily generated and acted upon by those we are trying to serve. One hundred sixty-one indicators will ensure that surely it is not.

My hope is that the dominance of quantitative statistical information as the sole, authoritative source of knowledge can continue to be challenged so that we embrace much richer ways of thinking about and understanding development.

***

This post originally appeared at: http://www.how-matters.org/2010/11/17/161-indicators/

***

Related Posts

Beyond the Ribbon Cutting

A New Discipline for Development Practitioners

Rethinking Trust, by Ben Ramalingam

More on Why ‘How Matters’

The Conundrum of Counting Beneficiaries

Views: 26

Tags: Change, Enough, Good, Guide, INGOs, Lesotho, M&E, Malawi, Mapping, Most, More…Outcome, RCTs, Significant, The, Zambia, aid, alternatives, assistance, building, capacity, community, development, evaluation, foreign, grassroots, indicators, international, logframe, monitoring, organizations, participation, philanthropy, workers

Comment

You need to be a member of Peace and Collaborative Development Network to add comments!

Join Peace and Collaborative Development Network

Comment by Sie.Kathieravealu on December 14, 2010 at 9:50pm

Thank you for the useful suggestion. I have already sent my ideas to them.

 

The ONLY way of tackling the problem of poverty is by changing the system of governance in each and every country that would empower and enable the POOR TO PARTICIPATE in the governance of the country.

 

My suggestions are towards achieving the above.

Comment by Jennifer Lentfer on November 18, 2010 at 11:43am
Thanks for your comments! @Landon. In my experience, it's been smaller, private donors i.e. foundations rather than public donors or aid agencies, which for obvious reasons perceive less freedom to explore alternative evaluation methodologies, including participatory action research. However, the very large and influential Gates Foundation of late, has embraced "living proof" of impact alongside management of real-time data. You can read more at: http://www.huffingtonpost.com/melinda-gates/share-the-proof_b_77203...
Comment by Elvira Mandic on November 18, 2010 at 10:52am
Great article! Thank you for sharing your knowledge and experience.
Comment by Landon Hancock on November 18, 2010 at 10:49am
Very interesting post and, although I can't consider myself an expert on development or evaluation, I agree with your assessment that funders (& projects) likely waste too much time and energy looking for and at quantitative indicators that may miss fundamental effects (or at least are too grounded in short-term outcomes to be helpful. I will look at some of the alternative methods you have links to, but some of what you're talking about sounds (a little) like participatory action research, which often has an evaluative component built in to it, generates a lot of rich qualitative (& often quantitative) data and is performed using many of the same indicators you point out. My question is whether, in your experience, funders (both large and small) are willing to fund projects that use these kinds of evaluation plans? Much thanks for the post.

Sponsored Link

Please Pay What You Can to Support PCDN

Please consider Paying What You Can to help PCDN grow. We encourage you to consider any amount from $1 and up. Read the SUPPORT page prior to making a payment to see PCDN's impact and how your payment will help.

Sponsored Link

Translate This Page



PCDN NETWORK TWITTER FEED

PCDN Guidelines and Share Pages

By using this site you're agreeing to the terms of use as outlined in the community guidelines (in particular PCDN is an open network indexed by Google and users should review the privacy options). Please note individual requests for funding or jobs are NOT permitted on the network.

Click BELOW to share site resources Bookmark and Share
or Share on LINKEDIN


FOLLOW PCDN on TWITTER, FACEBOOK or GOOGLE+

Google+

 

© 2014   Created by Craig Zelizer.

Badges  |  Report an Issue  |  Terms of Service