When doing review studies it quickly becomes apparent that methodology chapters have been reduced to descriptions of data collection, and occassionally, sometimes incidentally, a mention of analytical techniques in passing. Even the more experimental type work is frequently devoid of sufficient detail to judge its quality or reliability. If we were laboratory scientists replication would be out of question.

Methodolgy, I would argue, is first and foremost about research design. And research design is the framework of protocols and procedures adopted to ensure validity – internal first, external (a luxury or an impossibility depending on your epistemology) next. Frequently and conveniently this framework can be labelled (e.g. quasi-experimental, ethnographic) but most often, because we’re working in complex contexts, our framework is a mixture of methods that require an explict description and a convincing justification. When last did you read or write a methodology chapter that resembled anything like that?

To make sure I’m thorough in my descriptions of methodolgy I’ve developed a table that I populate while thinking through the approach to the evaluation. In the left column I enter the operationalised evaluation questions. In the next the sources of data for responding to those questions. The 3rd column is where I describe the data collection method for each data source. The 4th column describes the nature of the data – not just whether it’s qualitative or quantitative, but what sort of qualitative or quantitative e.g. ratio or interval or categorical or perceptions – suggesting parameters for the analytical techniques I can apply. The 5th column is a description of the analytical techniques applied to the data and a justification for the analytical choices I’ve made.

Of course challenging evaluation problems are seldom solved with a neat table, but it’s a tool to think through methodology systematically. All the while the persistent question, the constant refrain -”How is all of this addressing validity challenges?” And that ultimately is the question a thorough description of my methodology must answer – explicitly.

At FeedbackRA we take the response to our work personally. An evaluation exercise that fulfills a compliance imperative and decorates the bookshelf of some mid-level functionary (instead of bestowing upon the mid-level functionary powers of evidence to punch above her weight) erodes our sense of purpose, and we don’t like it. So we spend a fair amount of time thinking through how we can promote use. One of the more effective solutions (also feasible, unlike the emotionally charged calls for anonymous acts of intimidation – geeks have a dark side too) is composing a stand-alone policy brief that accompanies evaluation reports but that can also be distributed independently of the full evaluation.

A policy brief is a succinct summary of the implications of evaluation findings for decision-makers. The idea is to produce a single A4 page that focusses deliberations on evidence and guides decisions. The policy brief is structured as follows:

Conclusion
The brief leads with the objectives and evaluative statements or conclusions of the evaluation, as well as key lessons learned. To keep it short only the lessons with the mnost convincing body of substantiating evidence from the evaluation – in other words the findings representing the most reliable basis for taking action – are included.

Prioritize
This section of the brief highlights the most urgent calls to action indicated by the evaluation. These are often programming priorities but may be policy related if policy processes are already under way and policy level decisions are immanent.

Formulate
Policy recommendations are presented in this section. The intention of the policy brief is to facilitate evaluation use at levels of decision making beyond programme implementation. The crafting of this section is especially important. It must contain reasonable, unambiguous and evidence based recommendations that are not only convincing to the audience but are also posited by the evaluation team with a high degree of certainty.

Fund
If the evaluation provided evidence for either the most cost-effective spend, or ommissions in the programme design that would enhance effectiveness, then funding recommendations are approrpiate and are presented in this section of the policy brief.

Measure
An evaluation will frequently reveal gaps in the monitoring and evaluation framework developed for the intervention. If there is obvious adjustment to the M&E framework that would deliver significantly improved, utilisation focussed results, then those adjustments are proposed in this section of the policy brief.

Programme
Gaps in programme design, shortcomings in fidelity and the prerequisites for effective replication of the programme (should a finding on replication be required) are all addressed in a set of recommendations in this section.

Provided it is ruthlessly succinct, unambiguous, and firmly grounded in the stronger evidence provided by the evaluation, a policy brief is an indispensable weapon in the arsenal of the lowly evaluation stormtrooper bravely doing his daily duty at the frontline of the utility wars.

Greta Björk Gudmundsdottir wrote an interesting article in the open access journal: Internationl Journal of Education And Development: Using Information and Communication Technology. The article is titled: From digital divide to digital equity: Learners’ ICT competence in four primary schools in Cape Town, South Africa . It speaks to specifically computer skills which would be necessary for ICT solutions.

She says:
The potential of Information Communication Technologies (ICT) to enhance curriculum delivery can only be realised when the technologies have been well-appropriated in the school. This belief has led to an increase in government- or donor-funded projects aimed at providing ICTs to schools in disadvantaged communities. Previous research shows that even in cases where the technology is provided, educators are not effectively integrating such technologies in their pedagogical practices. This study aims at investigating the factors that affect the integration of ICTs in teaching and learning. The focus of this paper is on the domestication of ICTs in schools serving the disadvantaged communities in a developing country context. We employed a qualitative research approach to investigate domestication of ICT in the schools. Data for the study was gathered using in-depth interviews. Participants were drawn from randomly sampled schools in disadvantaged communities in the Western Cape. Results show that even though schools and educators appreciate the benefits of ICTs in their teaching and even though they are willing to adopt the technology, there are a number of factors that impede the integration of ICTs in teaching and learning.

For more on this, see Benita William’s M&E Blog