Something I frequently hear in evaluation recommendations is to focus more on the outcome level rather than outputs—the ‘so what?’ question. There are a range of ways to do this, one of which is called ‘Outcome Mapping.’
Hot Tip! “Outcomes are defined as changes in the behavior, relationships, activities, or actions of the people, groups, and organizations with whom a program works directly.”
Hot Resource: Check out Outcome Mapping: Building Learning and Reflection into Development Programs by Sarah Earl, Fred Carden and Terry Smutylo
Developed by the International Development & Research Centre in Canada, Outcome Mapping is an evaluative methodology that characterizes and assesses the contributions development programs make to the achievement of outcomes. It is especially useful in projects where success depends on behavioral or social change, and can also be used for project design. This blog, however, will examine OM as an evaluative methodology.
Hot Resource: Check out this introductory webinar to Outcome Mapping by trained OM facilitator Simon Hearn.
But first, let’s do a quick review of Outcome Mapping before looking at its evaluative application.
Outcome Mapping methodology is unique in that it shifts “away from assessing the impact of a program (which are defined as changes in ‘state’—for example, policy relevance or reduced conflict) and toward changes in the behaviours, relationships, actions or activities of the people, groups and organizations with whom a development program works directly.” The methodology does not ignore or suggest that changes in state are not desirable; on the contrary, OM states that in order for changes to be brought about in state, there must be changes in behavior which underline and support the state.
Hot Tip! “Boundary partners are those individuals, groups, and organizations with whom the program works directly and with whom the program anticipates opportunities for influence.”
Furthermore, Outcome Mapping states it is the boundary partners who are primarily responsible for bringing about change as a result of the intervention; the intervention merely facilitates the process “by providing access to new resources, ideas, or opportunities for a certain period of time.” Personally, I find this quite attractive: not only understanding how impact is brought about, but understanding how close we are in the causal chain to arrive at the desired outcome and/or impact.
Outcome Mapping uses ‘progress markers’ to understand progress towards the desired outcomes. Of course this assumes that you had properly identified outcomes in the design phase. We can apply this concept to evaluation, and construct progress markers ex-post based on our project objectives. To do this, we first must ensure that the objectives identify desired behavioural changes.
Hot Tip! Remember, Outcome Mapping is specifically concerned with changes in behavior, relationships, activities, or actions, so you may need to re-work your objectives a bit if this is not already the case.
For each objective, or ‘Outcome Challenge’ if you will, you should construct graduated progress markers that reflect the following levels:
You should additionally assign ratings for each of the progress markers; the rating should reflect the likelihood of the outcome occurring. For example, what we would like to see occur has a 70% projected chance of occurrence.
Hot Tip! All of this should be bound within the project timeframe: what can we realistically achieve, what would we like to achieve and what would we love to achieve within this timeframe?
The graduated progress markers you just set will now become part of the evaluative framework used to assess the project.
Hot Resource! Check out these case studies of how Outcome Mapping has been applied by IDRC partners.
Using a trained facilitator, ideally one who is familiar with Outcome Mapping, project staff together with Boundary Partners should begin to assess progress towards the Progress Markers. Using monitoring data, both that which is explicit and recorded and that which is intuitively felt, known or observed, the group collaboratively identifies which Progress Markers were achieved and which were not.
Based on these conclusions, the group can then reflect and identify challenges and trouble-areas for improvement in future programming. Outcome Mapping methodology is inherently geared towards learning.
Hot Resource! Check out Outcome Mapping, an online community of practice to support the use of OM methodology.
But there are several larger issues here that deserve discussion.
What should we be measuring when we do evaluation, the outcome or impact level? Being more in the so-called ‘circler camp’ (see Reina Neufeldt, “’Frameworkers’ and ‘Circlers’—Exploring Assumptions in Impact Asses...”), I find myself relating to the notion that social change is a long-term and complex process where attribution is not only difficult to draw, but that the process of doing so is inherently flawed as no single initiative can claim attribution for large scale changes frequently sought by peacebuilding initiatives. I therefore find OM quite appealing as it seeks to balance our need to measure and understand the changes which occurred within the project timeframe, thus keeping us more realistic in our goal setting—not just for ourselves, but for the people we serve; overpromising results can do harm.
And while I am not alone in the suspicion of attribution in peacebuilding, there are also trends in the opposite direction: the increasing use of impact evaluation as the ‘gold standard’ to be met, as well as a range of experimental and quasi-experimental methods that may not be as relevant to peacebuilding work both because of the difficulty in adapting the models (linear methodology for non-linear change processes) and the very real possibility of doing harm.
Regardless, however, the fact that both these trends are present is interesting and on the whole, I think, a good thing because it pushes us out of our measurement comfort zones (i.e., outputs), and hopefully at the same time, increases our effectiveness in peacebuilding activities—which ought to be the whole point of evaluation to begin with.
Outcome Mapping: Building Learning and Reflection into Development Programs by Sarah Earl, Fred Carden and Terry Smutylo
Outcome Mapping: Those Who Dream to Make a Difference by Sarah Earl, Raj Kumar Verma, Adama Ndiaye, Thierry Barreto Fernandes and Kalpana Pant
Introduction to Outcome Mapping Webinar by Outcome Mapping (outcomemapping.ca)
 Sarah Earl, Fred Carden and Terry Smutylo, Outcome Mapping: Building Learning and Reflection into Development Programs, Pg. 1.
 Sarah Earl, Outcome Mapping: Those Who Dream to Make a Difference, pp. 2.
 Earl, Carden, Smutylo, “Outcome Mapping,” pp. 3.