Two of the most important are...
- Evaluation is not marketing
- The function of measurement is improvement
abdi writes: Our position is that evaluation while deploying many of the tools of research, should not be treated as a research activity in itself. For all of the work we do (and the work of those we teach), the projects and programmes we and our students and clients are evaluating were commissioned and contracted on the basis that they were intended to achieve certain agreed outcomes. They were neither classified as research activities (and therefore funded as a research activity) nor were they setting out to test a theory or hypothesis. The funders (whether private, public or third sector) supported the project on the basis of the promised outcomes. No matter how vague and poorly specified, the basis of the project or programme agreement is ‘we are doing this in order to achieve that’.
If the project proposers do not know what the project outcomes might be, it is incumbent on them to undertake further research and analysis to identify the basis on which they claim ‘X will contribute to Y’ before they finalise the project proposals and agree the outcomes on which the project will be funded.
The first statement 'Evaluation is not marketing' makes it clear that the function of evaluation is not to find every and any possible way of telling the best stories. Over and again we find that organisations requesting evaluations are not at all clear about whether they really want to know what impact their project had achieved. Too often, they want to hear the good news and avoid collecting data that might show that while there was one great success story, there is no evidence to support whether the same impact was achieved with the other 99 out of the hundred beneficiaries or participating individuals or organisations. Or even, if there is data that suggests not everything was achieved, this data is either ignored or even in some cases, covered up. Of course, we recognise that it can be hard to present results data that suggest the project has either not achieved its outcomes or they are much less than planned. Evaluators that spend a lot of time with and get close to project teams and participants (for example those using a method of data collection and analysis known as participatory evaluation) risk losing independence and objectivity. But objectivity and independence are core characteristics of quality evaluation as well as fundamental to the need for the highest standard of ethics. If you take a look at some of the evaluation reports provided about public sector investments, it is a serious worry that so many seem to consider an evaluation an opportunity to carry out research (usually concluding that more research is needed) or worse still, report only the best possible picture of the project. Sadly, many of these are carried out by academic institutions that appear to have a different view of the function of evaluation than ours but because of the institution they represent, are often accepted by their commissioners as robust.
The second statement ‘the function of measurement is improvement’ is intended to demonstrate the importance of having a robust methodology (as opposed to method) which can provide the framework for classifying outcomes (as well as providing a process and agreed rules for measurement and analysis). Classifying outcomes within a structured framework enables the different types of outcomes to be measured for the purpose of improvement. For example, does everyone that needs to implement a new process, know what to do? Measuring the outcomes of an activity intended to provide the information required by project participants can help to ensure they are ready for the step of implementing a new practice.
This outcome framework supports what is sometimes called process evaluation. In other words, it helps identify what in the process contributed to the other outcomes, notably impact and value for money. It is also useful for analysing the effectiveness as well as the efficiency of each step in the process. This is evaluation as a management tool. Like all similar management tools that are used to improve, it ‘fits’ best in a culture that accepts that to improve you need good data to review, measure, reconfigure, re-measure and learn from in the continuous and virtuous cycle of improvement. Such ‘no blame’ cultures tend to be data driven collaborative and have a clear common shared vision of what they want to achieve and use measurement as an improvement tool.
If you would like to know more about how to build objective, independent rigorous evaluation capacity or would like to develop your competence in monitoring and evaluation and achieve a recognised UK professional and/or academic qualification, please contact us.
_____________________________________________