When doing review studies it quickly becomes apparent that methodology chapters have been reduced to descriptions of data collection, and occassionally, sometimes incidentally, a mention of analytical techniques in passing. Even the more experimental type work is frequently devoid of sufficient detail to judge its quality or reliability. If we were laboratory scientists replication would be out of question.

Methodolgy, I would argue, is first and foremost about research design. And research design is the framework of protocols and procedures adopted to ensure validity – internal first, external (a luxury or an impossibility depending on your epistemology) next. Frequently and conveniently this framework can be labelled (e.g. quasi-experimental, ethnographic) but most often, because we’re working in complex contexts, our framework is a mixture of methods that require an explict description and a convincing justification. When last did you read or write a methodology chapter that resembled anything like that?

To make sure I’m thorough in my descriptions of methodolgy I’ve developed a table that I populate while thinking through the approach to the evaluation. In the left column I enter the operationalised evaluation questions. In the next the sources of data for responding to those questions. The 3rd column is where I describe the data collection method for each data source. The 4th column describes the nature of the data – not just whether it’s qualitative or quantitative, but what sort of qualitative or quantitative e.g. ratio or interval or categorical or perceptions – suggesting parameters for the analytical techniques I can apply. The 5th column is a description of the analytical techniques applied to the data and a justification for the analytical choices I’ve made.

Of course challenging evaluation problems are seldom solved with a neat table, but it’s a tool to think through methodology systematically. All the while the persistent question, the constant refrain -”How is all of this addressing validity challenges?” And that ultimately is the question a thorough description of my methodology must answer – explicitly.

At FeedbackRA we take the response to our work personally. An evaluation exercise that fulfills a compliance imperative and decorates the bookshelf of some mid-level functionary (instead of bestowing upon the mid-level functionary powers of evidence to punch above her weight) erodes our sense of purpose, and we don’t like it. So we spend a fair amount of time thinking through how we can promote use. One of the more effective solutions (also feasible, unlike the emotionally charged calls for anonymous acts of intimidation – geeks have a dark side too) is composing a stand-alone policy brief that accompanies evaluation reports but that can also be distributed independently of the full evaluation.

A policy brief is a succinct summary of the implications of evaluation findings for decision-makers. The idea is to produce a single A4 page that focusses deliberations on evidence and guides decisions. The policy brief is structured as follows:

Conclusion
The brief leads with the objectives and evaluative statements or conclusions of the evaluation, as well as key lessons learned. To keep it short only the lessons with the mnost convincing body of substantiating evidence from the evaluation – in other words the findings representing the most reliable basis for taking action – are included.

Prioritize
This section of the brief highlights the most urgent calls to action indicated by the evaluation. These are often programming priorities but may be policy related if policy processes are already under way and policy level decisions are immanent.

Formulate
Policy recommendations are presented in this section. The intention of the policy brief is to facilitate evaluation use at levels of decision making beyond programme implementation. The crafting of this section is especially important. It must contain reasonable, unambiguous and evidence based recommendations that are not only convincing to the audience but are also posited by the evaluation team with a high degree of certainty.

Fund
If the evaluation provided evidence for either the most cost-effective spend, or ommissions in the programme design that would enhance effectiveness, then funding recommendations are approrpiate and are presented in this section of the policy brief.

Measure
An evaluation will frequently reveal gaps in the monitoring and evaluation framework developed for the intervention. If there is obvious adjustment to the M&E framework that would deliver significantly improved, utilisation focussed results, then those adjustments are proposed in this section of the policy brief.

Programme
Gaps in programme design, shortcomings in fidelity and the prerequisites for effective replication of the programme (should a finding on replication be required) are all addressed in a set of recommendations in this section.

Provided it is ruthlessly succinct, unambiguous, and firmly grounded in the stronger evidence provided by the evaluation, a policy brief is an indispensable weapon in the arsenal of the lowly evaluation stormtrooper bravely doing his daily duty at the frontline of the utility wars.

A sharper focus on equity in human development is gathering momentum at the international level, supported by United Nations reports and strategies as well as by independent analysis. Increasingly national policies and international alliances are focusing on achieving equitable development results. This emphasis poses important challenges and offers novel opportunities to evaluation.

“How can one strengthen the capacity of Governments, organizations and communities to evaluate the effect of interventions on equitable outcomes for marginalized populations? What are the evaluation questions to assess interventions are relevant and are having an impact in decreasing inequity, are achieving equitable results, and are efficient and sustainable? What are the methodological implications in designing, conducting, managing and using Equity-focused evaluations?”

 The new publication “How to design and manage Equity-focused evaluations” represents a first attempt to address these questions. The document starts by defining equity, why equity matters and why equity is so urgent now. It then presents what an Equity-focused evaluation is, explaining what its purpose should be and potential challenges in its promotion and implementation. The second part of the document explains how to manage Equity-focused evaluations, presenting the key issues to take into account when preparing for the Equity-focused evaluations and developing the Terms of Reference, including presenting potential equity-focused evaluation questions; how to design the evaluation, including identifying the appropriate evaluation framework, evaluation design and appropriate methods to collect data; and how to ensure the evaluation is used. The document also addresses how to conduct Equity-focused evaluations under real-world constraints. Last but not least, eight case studies are included to illustrate how evaluations supported by UNICEF have addressed equity-focused issues.

 The new publication is available, free of charge, at  

http://www.mymande.org/?q=virtual  .

 An introductory webinar with the authors, and a PowerPoint presentation, are also available at http://www.mymande.org/?q=how_to_design_implement_and_use_evaluations