Feedback is an innovative research, evaluation and information quality services provider, inspired by the challenges of sustainable development. By applying creativity, conscience, obsessive curiosity and uncompromising scientific rigor, we empower clients with the best possible empirical basis for taking action and making an impact.

Articles

0

Missing the Big Picture

by Terence Beney
October 18th, 2012

image

A productivity evaluation of Schubert’s Unfinished Symphony

1. For a considerable period, the oboe players had nothing to do. Their number should be reduced, and their work spread over the whole orchestra, thus avoiding peaks of inactivity.

2. All twelve violins were playing identical notes. This seems unnecessary duplication, and the staff of this section should be drastically cut. If a large volume of sound is really required, this could be obtained through the use of an amplifier.

3. Much effort was involved in playing the demi-semiquavers. This seems an excessive refinement, and it is recommended that all notes should be rounded up to the nearest semiquaver. If this were done, it would be possible to use trainees instead of craftsmen.

4. No useful purpose is served by repeating with horns the passage that has already been handled by the strings. If all such redundant passages were eliminated, the concert could be reduced from two hours to twenty minutes.

In light of the above, one can only conclude that had Schubert given attention to these matters, he probably would have had the time to finish his symphony.

Articles

0

The Future of Evaluation

by Terence Beney
October 4th, 2012

image

Gargani and Company have started a blog on the future of evaluation, where you can post predictions. It makes for interesting reading. Here are my contributions:

1. As data becomes ubiquitous data mining will become a crucial skill. However the need to generate primary data in especially the developing world, where large gaps in routine data remain, should see the traditional tasks of evaluation continue. Will we perhaps see sharper differentiation in the profession between data specialists (data managers and miners) and researcher-evaluators (who generate new data as part of their process)?
2. Surveillance technology will become an increasingly important source of data, driven by the (once again) ubiquity of routine surveillance data, a market that demands it, and the public’s willingness (or resignation) to sacrifice privacy while preserving anonymity.
3. Evaluation reports will not be the primary output of evaluators. Instead it will be clean, collated data sets and bespoke analytical tools (algorithms) for clients to make the most of those data sets. Evaluators will still provide analysis, but not frequently in the traditional report format.
4. Analytical outputs will become more concise and more visual. This will parallel our technology augmented ability to apply systems analytical techniques to the increasingly complex evaluands we will be required to evaluate.

What won’t diminish (here G&C have it wrong) is evaluation theory. Theory is as much about ontology as epistemology. Ontology is ideological and ideology is impervious to proof or even reason – so just because a set of techniques consistently produces sound evaluation results that does not mean that theory will be usurped by some framework grounded in techniques that work. Theory is more likely to proliferate than diminish. Practitioners however will continue to be pragmatic about theory and employ what works.

Articles

0

Social Net Present Value: The Bottom Line

by Terence Beney
March 25th, 2012

20120325-232006.jpg

Presentations at the recent Mining Indaba on using Social Net Present Value (SNPV) methods to report on corporate social investments and responsibility once again highlighted the risks of resorting to an accounting framework to measure impact. The highlighting was done by those remarking from the floor, not the presenters who appeared intent on selling a version of the methodology to CSR practitioners desperately seeking for heuristics to translate development language into corporate speak.

Some intensive work has been done on SNPV methodology and it is not without merit. But SNPV is by no means the panacea to impact measurement complexities that some tool peddlers insist. There are at least 4 areas that problematise SNPV as method:

1. SNPV Light. While the theoretical work on SNPV adopts a proper accounting formula to arrive at a number, CSR in practice does not mirror that level of rigour. I have yet to see a publicly available corporate report that values the enterprise’s social benefit AND social cost, then subtracts cost from benefit to arrive at a result. In addition some methodologies rely on the dubious technique of extrapolating impact from at the expense of actually collecting time series data, an astonishing error blatantly entertained by ‘service providers’ offering SNPV solutions at the conference.
2. The complexities of assigning value OR the reductionist error. This aspect alone is grist for multiple contending journal articles, but for the purposes of the post lets distill the toxic mix to the most self-evident point: how do we arrive at valid quantitative values for the outcomes of social investments? How do you value something like improvements in education or health services satisfactorily? How do you assign a single quantitative or monetary value to community cohesion resulting from improved social housing and reductions in migrant labour? And do so credibly?
3. The equivalence problem. Presumably one of the attractions of SNPV is its purportedly efficient aggregation of social investment performance to a single value. This implies however that somehow a quantitative method must be formulated that can validly assign equivalent units of value to say ‘improved employability prospects as a result of education programmes’ and ‘improved productivity as a result of health programmes’, and then add those values together for a net result. Anyone who grasped basic number theory in under-graduate statistics should be inspired to new heights of skepticism right here.
4. Perversely incentivized social responsibility. While we don’t have an instance in mind, behavioral economics will tell you its plausible: if there is a significant discrepancy between SNPV from health over education programming, then loading our CSI portfolio with health projects makes for a better bottom line performance. And this may become a criteria for investment decisions that trumps community needs. Development prioritized by corporate performance objectives . . . Does the phrase ‘short-termism’ come to mind?

SNPV has its uses, but social phenomena are complex and the constellation of social phenomena and processes constituting ‘development’ is inordinately complex. It has no single bottom line impact, at least nothing that will stand up to the scrutiny of the basic test of measurement credibility in social science – validity. In development there are many lines. And not just at the bottom. Everywhere.

20120325-232006.jpg