What makes an evaluation influential? How can we ensure that it is read, understood, and used in decision-making? For this article, we interviewed a well-known expert in evaluation, Burt Perrin. In it, he discusses why evaluations need to be influential and what we can do to make sure they have an impact on programs, policies and strategies of organizations.
Q: What should we consider an influential evaluation? How do you measure its influence?
An influential evaluation is one that contributes, in one way or another, to improvements in people’s lives. A good starting point here is the African Development Bank’s (AfDB) Mandate. Its objectives are to achieve inclusive growth, deep reductions in poverty, an increase in jobs, and equality of treatment and opportunity. From this starting point, evaluations should influence the directions of policies and programs towards these objectives.
Beyond this, it is important for evaluations to be as meaningful as possible, to think beyond strategies and look at impact, the changes in peoples’ lives, in accordance with the AfDB’s mission and overarching objectives.
Assessing the extent of the use or influence of an evaluation, however, can be tricky. There is extensive literature on what is called ‘evaluation utilization’, which looks beyond the direct use of evaluation at other ways in which evaluation leads to different forms of thinking and approaches.
Evaluations can be both formal and informal, so understanding their influence is complex. One way of looking at it is the Theory of Change approach, to look at the entire process of evaluation, including its use by the institution, and to see if people are really better off at the end of it.
Q: In some institutions, if not many, evaluators find that people are not always very open to the evaluation process and it is seen as quite a formal process. Can you give an example of an informal approach that can be influential?
An evaluation can be very formal, but its use can be informal. In the interaction between people involved in projects and people involved in evaluation, for example, there may be opportunities to share insights on how things can be done differently. This is in addition to the final evaluation report that you end up with. Thus, learning can be informal where going through the process of an evaluation can be instrumental in helping to create learning within an institution like the Bank.
Q: Why is it important that evaluations are influential?
Evaluation by itself has no intrinsic value. It only has value if it is used in some way to help to make a difference. Indeed, if it is not used in some way, an evaluation can even be negative, for example consuming human and financial resources.
Furthermore, there are no easy answers to most development challenges. Since there is no such thing as the perfect program or strategy, we look at the extent to which the right approaches are being used to address the challenges faced and whether they can be improved in some way. Evaluation has the potential to be an important resource, a tool to learn what is working well and why, by contacting and getting insights from beneficiaries for example and to help identify ideas about what else might be done or might be done better.
For example, the AfDB’s mission to achieve sustainable, inclusive and equitable growth that leads to poverty reduction is challenging. Evaluation has the potential to play a significant role in helping the Bank. No single program will achieve these things by itself, but by learning from one piece of the puzzle you can help make the other ones work better.
It is important to bear in mind that evaluation is often viewed as a negative thing, something that identifies problems. But evaluation can and should also identify what is working well and this can validate good practices that should be continued or even expanded, and can also help generate energy in being even more effective.
Q: What are the factors that contribute to evaluations’ influence?
The first and arguably most important factor is the need to think about impact from the very start. Too often the evaluation process is focused on developing a report and going through the motions. But really thinking about how your evaluation impacts people’s lives sets the stage for impact and influence. Often, without thinking in those terms it just doesn’t happen.
Secondly, we need to recognize that meaningful change rarely happens just as a result of a single intervention. It’s important that evaluators look at what they can and can’t accomplish, bearing this in mind.
Another important factor that contributes to influential evaluations, is taking a positive and constructive approach. This doesn’t mean ignoring the things that are evidently not going well in a project, but rather to frame these in a constructive context, such as what could be done more or better. Furthermore, we need to document what is working well because we need to also be looking at what we can do, not just what we shouldn’t do.
Another important way to make evaluations influential is to actively engage key stakeholders throughout the evaluation, the intended users, other stakeholders such as Regional Member Countries and civil society, and intended beneficiaries of the projects or strategies being evaluated. It is helpful and important to hear their perspectives. This engagement needs to involve far more than sharing terms of reference or draft reports, or asking for comments or just interviewing people. It needs close collaboration throughout the entire evaluation process. This can include for example joint discussions on the focus of the evaluation, which questions should be answered, or also on emerging findings and potential implications.
Having credible evidence for the different stakeholders who might be using it, contributes to evaluations’ influence, as well as adding voice to the voiceless. An evaluation can show how strategies affect people for whom they seem pretty remote.
Engagement doesn't need to compromise the independence of the evaluation. Ultimately, the evaluator has an obligation to tell the truth, but invariably through discussing these things, an approach can be found that everyone can live with. Engagement throughout the evaluation process can help develop ownership in evaluation, with stakeholders more likely to be interested in investing time and energy in acting upon it. An active approach such as this requires an investment of time but there is the potential of a strong payback.
Flexibility and responsiveness are also very important. Programs do not operate in a static environment and things are changing at a rapid pace. The COVID-19 pandemic is a great example of how we just can’t assume the ongoing appropriateness and relevance of objectives, indicators and targets set in advance. As the context changes, new opportunities inevitably emerge as well. It is important that the evaluation responds to the changes in the program and adapts to its context.
For evaluations to be influential, it’s also important to explore explanations and generate an understanding of why something has been happening or not. You cannot inform future directions without knowing why.
|
On evaluation for accountability vs. for learning: We often view accountability and learning as polar opposites. But it depends on how we approach accountability. Asking difficult questions and acting upon these, not only doing evaluation but also using evaluation to improve, these are really high standards to have. Accountability in this way is part and parcel of learning. |
Q: What kind of a relationship and collaboration are needed between the evaluator and the evaluand?
It is about establishing a collaborative process, not just sharing TORs. Sometimes evaluation is seen as a necessary evil, something to endure. However, if it is viewed and approached in the right way, it can represent a real asset and an opportunity, which in addition to providing ideas about program effectiveness, can also identify barriers outside the program’s direct control.
Ideally, there needs to be some form of partnership process where the program being evaluated and the evaluator are working together to enhance the value of the evaluation, to make it as meaningful as possible. This way the evaluation can be viewed as a tool that can help programs improve, to learn to better address the needs of the intended beneficiaries.
It is not just the evaluator who has an important role to play. Evaluation is too important for evaluators to do just by themselves.
We found that looking at those evaluations that have made a difference, invariably there was someone to champion it. Sometimes this was someone at a senior level who can encourage his or her staff to support the evaluation, or sometimes it was someone part of the program that became a natural champion. Users are also the ones that can embed evaluation and evaluation thinking within the program. Finally, the users of an evaluation often have better ideas than the evaluators in terms of how best to engage with key stakeholders, improving the influence of an evaluation.
Therefore, engagement is key. The benefits can flow both ways: On the one hand, the right engagement provides an opportunity for those involved in what is being evaluated to influence the direction of the evaluation, ensuring that it is asking the right questions, making it more relevant. On the other hand, the engagement also increases the ownership by key stakeholders, making them much more likely to act upon the recommendations.
What can also follow from the engagement of users in the evaluation process is referred to as process use. One often thinks of an evaluation exercise as ending with someone reading a final report, looking through the recommendations and the lessons learned. However, there is extensive literature that says that how an evaluation is done is an important factor in how it is used. When users are engaged in the evaluation process, they often can gain insights well before they are listed in a final report, as well as other small details that may not make it into the report.
Q: At the AfDB Evaluation Week, you will interact with young and emerging evaluators, evaluators from the African Development Bank as well as the global evaluation community. What will your message be, thinking about what we discussed today?
I want to leave you with three key points to remember:
Firstly, we need to recognize the strong potential for evaluation to be influential, to make a difference and to play a contributing role to better programs, policies and directions – and for improved lives of people living in Africa.
Secondly, this influence is not automatic – there are many things that an evaluator can do to help make an evaluation useful and influential.
Thirdly, there are key roles, not just for evaluators but also users of evaluation. These roles could be to help support influential evaluations that in turn can help them enhance the impact of what they are trying to do.
For more information
For more ideas about how evaluation can help to make a difference, see the publication Evaluations that Make a Difference: Stories from Around the World, available in both English and French at https://evaluationstories.wordpress.com/evaluation-story-publications/
Burt Perrin’s most recent publication: Changing Bureaucracies: Adapting to Uncertainty, and How Evaluation Can Help has just been published by Routledge. For more information go to https://bit.ly/3353qJk
Burt can be contacted at Burt@BurtPerrin.com