“The most insightful research on development actors end with a fight”
“The most insightful research on development actors end with a fight” (Dirk-Jan Koch)
As university trained researchers we recently made the shift to applied research in the development sector. We now work at an NGO and research institute in which we evaluate the impact of development projects. However, we quickly came to understand, the practice of doing research is not the same at our current jobs as it was at the university. In his morning lecture, Dirk-Jan Koch explored these differences with us. And the big question remained: are evaluation studies in the development sector completely independent?
What is evaluation research and why are evaluations important?
It is challenging to give a clear definition of what an evaluation exactly is. However, a widely used and broad definition for evaluation is the following: “Something is assessed by someone to some criteria in some way” (Kromrey, 2001). There are several reasons to conduct an evaluation, two important reasons are: (1) to learn from projects and develop them (2) to legitimate spending and ensure funding for future projects.
Although development projects are designed to achieve positive impact, the reality is not always as straightforward. Often, these goals are not fully achieved and there might even be negative (side) effects. Some researchers even argue development aid is dead and that it is the “White Man’s burden” that prevents poor countries from developing. Other, more optimistic researchers, claim that one should continue with development work, but urge that we need to understand what kind of aid works and what doesn’t. Basically, they say that we should focus more on the evaluation of development programmes. According to this latest, evaluation studies can help programmes to become the better version of itself and create more positive impact.
Secondly, evaluation studies are used by many organisations to legitimate spending and ensure future funding. Many of the development organisations we (AMIDs) are working for, are part of a Strategic Partnership in the Dialogue & Dissent framework of the Ministry of Foreign Affairs. Most of the current partnerships hopes to get funding for a second term. One could imagine that the Ministry only wants to continue with those organisations that provide convincing “evidence” of the positive impact they make with their programmes. If they can’t, how could the Ministry legitimize to the Dutch taxpayer that money is spent on these programmes? To understand whether the organisation reached their objectives, the Ministry requires that “Each partnership will present an independent evaluation”. In practice this means that the partnership will involve independent researchers to evaluate their programmes on various criteria. For example, on effectiveness, efficiency, relevance and sustainability.
Is evaluation research independent?
The optimists will say ‘yes’ and refer to the Netherlands Code of Conduct for Research Integrity (link). This “code” consists of five core principles: Honesty, Scrupulousness, Transparency, Responsibility and Independence. This code enforces researchers in the Netherlands to stay true and independent while evaluating the programmes.
The pessimists, or perhaps we could say realists, say ‘no’. They refer to a conflict of interest and the power relation between the evaluation researchers and their employers or clients. Often, the NGOs that order a study would benefit by a positive result. It shows (a) that they are doing a good job and (b) it would help them to get additional funding. Hence, they can try to stir studies by setting the research scope, decide on the research team, often hold a key position to access information, decide on who approves the research report whether it will be published or not.
Researchers, both at an independent organisation as in-house researchers, often have conflicting interests in these situations. On the one hand they want to work according to the Code of Conduct for Research Integrity, but on the other hand, they want to please their employer or client. Hence, in their decision they need to take the code of conduct of their organisation into account. It often includes principles such as “Being on the payroll of a development organisations means that one is expected to uphold the public image of the organisation, contribute to a continuous flow of funding to the organisation and continuously improve the operations of the organisation.” Not to forget that at the end of the year a decision needs to be made about renewal of their contract or being hired for a new research project. Hence, a loyal employee could be tended to contract a researcher with whom the person had good experience in the past. Researchers who have been too critical in the past aren’t very likely to obtain a second assignment. Also, being too critical and too much focussed on the negative effects could make the organisation decide to end the cooperation, meaning that the researcher will not get its hours paid.
So, independent evaluation research is not possible?
We absolutely did not say that. It is possible, but the risks that are described above should be tackled. This means especially that organisations who commission an evaluation study should have the full intention to get a grip on the reality and learn from the findings. This means that researchers should be totally free to come up with a critical report, without jeopardizing their own position.
That also means that there should be a mind shift within the entire development sector. Currently, organisations tend to share mostly their successes and hardly share their mistakes and lessons learned. This is reinforced by the funding structure that makes an organisation vulnerable in case evaluations do not report the envisioned impact.
So currently, we observe that when conducting independent evaluations, this often causes friction between the researchers and their clients. In many cases researchers are to some extend controlled by their clients, which influences study outcomes. However, it is good to remember that the most insightful studies on development programmes end with a fight.
Kromrey, H. (2001). Evaluation-ein vielschichtiges Konzept: Begriff und Methodik von Evaluierung und Evaluationsforschung; Empfehlungen für die Praxis. Sozialwissenschaften und Berufspraxis, 24(2), 105-131.
Written by Jonne Bosselaar (WEcR) and Raymond van Teeffelen (RNW Media) on the lecture on Impact Assessment by prof. Dirk-Jan Koch