When doing review studies it quickly becomes apparent that methodology chapters have been reduced to descriptions of data collection, and occassionally, sometimes incidentally, a mention of analytical techniques in passing. Even the more experimental type work is frequently devoid of sufficient detail to judge its quality or reliability. If we were laboratory scientists replication would be out of question.

Methodolgy, I would argue, is first and foremost about research design. And research design is the framework of protocols and procedures adopted to ensure validity – internal first, external (a luxury or an impossibility depending on your epistemology) next. Frequently and conveniently this framework can be labelled (e.g. quasi-experimental, ethnographic) but most often, because we’re working in complex contexts, our framework is a mixture of methods that require an explict description and a convincing justification. When last did you read or write a methodology chapter that resembled anything like that?

To make sure I’m thorough in my descriptions of methodolgy I’ve developed a table that I populate while thinking through the approach to the evaluation. In the left column I enter the operationalised evaluation questions. In the next the sources of data for responding to those questions. The 3rd column is where I describe the data collection method for each data source. The 4th column describes the nature of the data – not just whether it’s qualitative or quantitative, but what sort of qualitative or quantitative e.g. ratio or interval or categorical or perceptions – suggesting parameters for the analytical techniques I can apply. The 5th column is a description of the analytical techniques applied to the data and a justification for the analytical choices I’ve made.

Of course challenging evaluation problems are seldom solved with a neat table, but it’s a tool to think through methodology systematically. All the while the persistent question, the constant refrain -”How is all of this addressing validity challenges?” And that ultimately is the question a thorough description of my methodology must answer – explicitly.