Feedback is an innovative research, evaluation and information quality services provider, inspired by the challenges of sustainable development. By applying creativity, conscience, obsessive curiosity and uncompromising scientific rigor, we empower clients with the best possible empirical basis for taking action and making an impact.

News & Events

0

National Evaluation Policy Framework

by Terence Beney
December 8th, 2011

SA Coat of Arms

The Department of Performance Monitoring and Evaluation in the Presidency has just released the National Evaluation Policy Framework. As the introduction to the document explains:

“The Policy Framework for the Government-Wide Monitoring and Evaluation System (GWMES) which was approved by Cabinet in 2005 describes three “data terrains” which underpin the monitoring and evaluation system, namely, programme performance information; social, economic and demographic statistics; and evaluation. While the Presidency is the custodian of the GWMES as a whole, National Treasury has published the Framework for Programme Performance Information and Statistics South Africa has published the South African Statistics Quality Framework to provide policy frameworks for the first two terrains. This National Evaluation Policy Framework completes the set of policies which make up the GWMES.”

With the publication of the Framework the GWMES now has a complete set of guidelines and the foundation for consistent assessment of government’s service delivery performance has been laid.

The framework is downloadable from the Presidency’s website or follow the link National Evaluation Policy Framework.

The guidelines for the additional two terrains of performance assessment can be downloaded by following the links in the text above, or by visiting the StatsSA and National Treasury websites.

News & Events

0

Evaluation vs.Social Research

by Terence Beney
December 1st, 2011

jumping-brain

John Lavelle’s diagrammatic representation of the difference between Evaluation and Social Research. What’s missing, I would suggest, is USE. Evaluation is driven by an intention to use research results immediately. Research is concerned with generating a body of knowledge but is not necessarily preoccupied to the degree evaluation is with Use issues. That is not to say that Use doesn’t matter to social researchers – but Use is not usually a primary objective for Research to the extent that it is for Evaluation. At times it may even be an incidental consideration.

Does that not make Evaluation an applied social research practice? And where would you put USE on the hourglass?

Articles, News & Events

0

Writing Methodology: It’s Not Data Collection

by Terence Beney
November 30th, 2011

methodology

When doing review studies it quickly becomes apparent that methodology chapters have been reduced to descriptions of data collection, and occassionally, sometimes incidentally, a mention of analytical techniques in passing. Even the more experimental type work is frequently devoid of sufficient detail to judge its quality or reliability. If we were laboratory scientists replication would be out of question.

Methodolgy, I would argue, is first and foremost about research design. And research design is the framework of protocols and procedures adopted to ensure validity – internal first, external (a luxury or an impossibility depending on your epistemology) next. Frequently and conveniently this framework can be labelled (e.g. quasi-experimental, ethnographic) but most often, because we’re working in complex contexts, our framework is a mixture of methods that require an explict description and a convincing justification. When last did you read or write a methodology chapter that resembled anything like that?

To make sure I’m thorough in my descriptions of methodolgy I’ve developed a table that I populate while thinking through the approach to the evaluation. In the left column I enter the operationalised evaluation questions. In the next the sources of data for responding to those questions. The 3rd column is where I describe the data collection method for each data source. The 4th column describes the nature of the data – not just whether it’s qualitative or quantitative, but what sort of qualitative or quantitative e.g. ratio or interval or categorical or perceptions – suggesting parameters for the analytical techniques I can apply. The 5th column is a description of the analytical techniques applied to the data and a justification for the analytical choices I’ve made.

Of course challenging evaluation problems are seldom solved with a neat table, but it’s a tool to think through methodology systematically. All the while the persistent question, the constant refrain -”How is all of this addressing validity challenges?” And that ultimately is the question a thorough description of my methodology must answer – explicitly.