This publication should be cited as: Engel, P., Carlsson, C. and A. van Zee. 2003. Making evaluation results count: Internalising evidence by learning. (ECDPM Policy Management Brief no. 16). Maastricht : ECDPM

Development agencies are consistently required to improve their performance, in terms not only of project outcomes, but also of the quality of their programming and their institutional capacity. In a very practical sense, evaluations are now perceived as learning opportunities. It only seems logical, therefore, to try and improve the internalisation of evaluation results at different levels. But how can this be done? What dilemmas do we face and what have we learned so far?

This brief draws substantially on a recent workshop entitled 'How can we learn more from what we do? Evaluation and evidence-based communications for development'. The workshop was organised jointly by the ECDPM and Exchange, and was held in Maastricht on 13-14 December 2002. Its findings were also presented to the DAC Working Party on Aid Evaluation Workshop in Paris in March 2003.

When will we ever learn ?

'There is a real sense of urgency about the need to reinvent our institutional frameworks.' (Maastricht workshop participant)

Development practice being a quest for innovation and societal change, there is clearly an ongoing need to find out exactly what works when and in which circumstances. Against this background, it is becoming increasingly common to build evaluations into processes of institutional learning. Although partnerships, national ownership, facilitation, stakeholder participation and dialogue are all important aspects of evaluation, as well as a marked tendency towards the decentralisation of decision-making, effective communication is also becoming more and more important as an integral part of the evaluation process.

Evaluation¹ has always been about learning, about how to be accountable, how to be transparent, how to learn from experience. The issue today, therefore, is not whether it is desirable to learn. Instead, the question is: who should learn, why should they learn and how should they set about it? Quality issues also come into play if learning is to be widely shared.

Spearheaded by the OECD/DAC Working Party on Aid Evaluation (2001), donors have begun to intensify their learning efforts and improve their practices. Within this context, we should not take too narrow a view of the need for improving the tools of the trade. We need to make a serious effort to open up new vistas, refine and develop new approaches, devise new tools and inquire into new experiences. In our contribution to this debate, we propose to look at three strands of current thinking on improving learning in development, each at a different level of societal complexity, and try to match these with evaluation practice.

Internalising evidence from evaluations: three perspectives on learning

(a) In development policy and programming

'Donors must enhance their capability for undertaking continuous learning, and use techniques that allow partners and stakeholders to act as the primary sources of information on changing opportunities and institutional capacity constraints.' (Donor representative, Maastricht workshop)

Probably the most straightforward way of enhancing learning is to look at its internalisation in development policy and programming. However, many of the existing feedback mechanisms are still mainly one-directional, drawing on the logic of information dissemination to selected target groups rather than communication around evidence as a reiterative learning process.

The OECD/DAC Working Party on Aid Evaluation (2001) reviews some of the current experiences. Most reports have called for the establishment of what may be called policy innovation networks, mobilising Southern partners and stakeholders to engage in existing learning and feedback routes. The International Fund for Agricultural Development (IFAD) recognises the need to 'shift the fulcrum of evaluation feedback to the South' to encourage more direct links between feedback of findings and the planning and monitoring of country programmes. Many development agencies emphasise results-based planning and management as a way of improving the practical use of evaluation results. The European Commission's 'Fiche Contradictoire' and UK's Department for International Development's (DFID) 'Public Service Agreements' are examples of this. The review regards the communication of lessons through the mass media as increasingly important. Finally, the review calls on development agencies to do more to share their experiences with each other.

(b) In organisations and among partners

'Evaluations of organisations are often used as 'frustrated PR tools' … to demonstrate impact; not necessarily to be self-critical and to learn how to improve performance.' (Evaluator, Maastricht workshop)

A second approach focuses on organisational learning, recognising that development processes result from actions and interactions by a set of diverse stakeholders. Active participation, capacity-building and learning by all these actors is a fundamental rather than an instrumental condition. The locus for change is the facilitation of collective rather than individual learning. As a result, policy-makers and/or donors become one among many, rather than the only intended learner.

An organisational learning approach to evaluation not only fundamentally changes the way social actors relate to each other, it also requires a radical shift in the role of the evaluator. All actors, including the evaluator, have to recognise they are part of a joint learning effort. In such an 'epistemic community', the evaluator becomes a facilitator in a joint inquiry rather than an expert wielding an 'objective' measuring stick. Yet such communities run the risk of 'clique-building', reducing the diversity of opinion if the discourse is captured by the most vocal actors in the group (Sutton, 1999). Critical self-reflection must be maintained in order to build in 'reality checks' and thus avoid too narrow a discourse among closed circles.

Box 1: CIET social audits: socialising evidence for participatory action (SEPA) The Community Information and Epidemiological Technologies (CIET) social audit methodology aims to build the community voice into the evaluation process. It uses a 10-step approach to gather relevant local data that can be used for social action and planning. A crucial step in the method is the enrichment of evidence with local stakeholder views and the sharing of results with service-providers and users by means of workshops and training sessions. CIET evidence is 'socialised' through advocacy networks, the mass media and key opinion-makers like political and religious leaders, who are trained to make sense of the evidence for local action. The feedback loop from data gathering to information sharing for policy, planning and local action is fairly rapid and, repeated periodically, fuels an awareness and discussion about public service performance, helping to ensure that planners and donors are held accountable to beneficiaries. For more information, see: Robert J. Ledogar & Neil Andersson, Social Audits: Fostering Accountability to Local Constituencies, in: Capacity.org, ECDPM, Maastricht: issue no.15., or www.ciet.org

(c) In society at large

'[Societal learning] requires a fresh attitude to evaluation findings, and encourages a multi-stakeholder debate on evaluation results …' (Maastricht workshop participant)

A third perspective focuses on a type of learning that leads to change in society at large. When the sharing and interpretation of evidence extend beyond those directly involved in the evaluation process, conflicts of interest are common and consensus becomes the exception rather than the rule. The question then is whether and how interested parties can exert pressure for change and whose interpretation of the findings is the dominant one. The co-existence of multiple truths requires a more transparent analysis of findings and the creation of 'sense-making fora' for stakeholders to interpret and validate the evidence. Some commentators stress that such broadening of the interpretation of evidence to a wider audience and different interest groups can help to avoid 'paradigm traps' among scientists and policy-makers that limit their views on development options (Uphoff, Combs, 2001).

Linked to the societal uptake of evidence is what Weiss described as 'knowledge creep' (1980), i.e. the way in which the conceptual use of evidence can 'gradually bring about major shifts in awareness and reorientation of basic perspectives' among a broader audience. These ideas have recently re-emerged in concepts such as knowledge management and knowledge as a global public good, pioneered by the World Bank and others. Yet the use of scientific evidence - derived from evaluations, research or other country analytic work to address development problems - has long brought with it the risk of failing to connect with realities and evidence at a local level and with methods that seek to record and enhance endogenous development processes and knowledge. On the other hand, the knowledge systems approach pioneered by Wageningen University in the Netherlands focuses on innovation as an emergent property of social interaction and learning among multiple stakeholders who, invariably, represent multiple intentions and (often conflicting) interests (Röling, 2002, Engel and Salomon, 1997).

The perspective on societal learning from evaluations, i.e. learning that goes beyond a small number of directly involved stakeholders, could have far-reaching consequences for our thinking on development cooperation. Traditional evaluation as we know it may gradually fade into the background, to be replaced by multiple forms of evidence-gathering and sharing among diverse groups of stakeholders, the adaptive management of resources and multiple communication and negotiation processes at various levels. There will be a greater need for good governance to create enabling conditions for such processes and for conflict resolution between stakeholders. As a result, governments and donors will become crucial players who may either enable or prevent society from learning. As both development policy-makers and field practitioners alike have urged, this perspective links evaluation with governance issues.

Dilemmas at the evaluation-learning interface

'Local and global events produce emergent and unpredictable outcomes through cascades of feedback. Social change is not a one-off activity … Are we prepared to listen?.' (Maastricht workshop participant)

Various dilemmas arise in trying to turn the current evaluation practice into a learning process.

• Why strengthen the learning function? What other purposes should evaluation serve?

Evaluation serves a number of different purposes (see diagram). A growing emphasis on learning from evaluation means a shift in intentions. Traditionally, control has been an important purpose. From this perspective, the aim of evaluation is to enhance transparency and accountability, particularly from the donors' point of view. Another vital purpose is assessment, i.e. deciding whether the agreed objectives have in fact been achieved. With learning in order to improve performance now becoming an increasingly important purpose, evaluations could eventually become geared towards adaptive management. Institutional learning and the development of the institutions responsible for managing development would be prerequisites for this.
So would pursuing one purpose imply a trade-off with another? Might these aims be mutually exclusive in an evaluation context? Or would a shift in intentions from quadrant one to four reflect a growing complexity of evaluation functions rather than a shift from one to the other?

Assessment

II Learning	I Control
III	IV

Adaptive Management

• What do we mean by 'learning'?

Learning is a buzz-word, often used but not often clearly defined. As a result, one of the challenges defined by the OECD/DAC Working Party on Aid Evaluation 2001 is the need to unpack the learning concept. After all, there are all sorts of different approaches. In adult education, different types of learning are linked to distinct cognitive and motivational processes and fulfil complementary roles (Van der Veen, 2000). Organisational learning theory distinguishes single-, double- and triple-loop learning, reflecting the degree to which underlying rules, values, norms and behaviour are truly affected (King & Jiggins, 2002). A third approach focuses on cognition, the process by which an organism deals with changes in context. It stipulates two fundamental drivers for social learning: the coherence sought among values/emotions/perceptions on the one hand and theory/interpretations/actions on the other; and the need for correspondence between the above elements and the prevailing context (Röling, 2002).

Box 2: The DFID experience of moving to thematic networks for learning To foster a more enabling climate for organisational learning, a Knowledge Sharing Consultation Group was established within the Rural Livelihoods Department of DFID, UK. The group examined various aspects of knowledge-sharing, such as cultural issues, content, processes and technology. It found that there are many disincentives to sharing knowledge: team and individual behaviour tends to emphasise competitiveness rather than sharing; learning is inward-looking, instead of being based on external performance checks; and technology is used predominantly for 'information provision' rather than knowledge-sharing'. As a result, the group formulated its mission for 2003-2004 as 'linking ideas and knowledge with those who will benefit from them'. It identified a number of focal issues: improving staff appreciation of methods of knowledge-sharing; clarifying the terminological confusion; linking knowledge-sharing and learning directly into organisational strategy; and identifying champions of organisational learning. One of the main conclusions drawn by the group is that knowledge is shared differently in different working environments, and in relation to the operational modes used within these environments. The key factors in this respect are the degree of teamwork as opposed to individual work, and the proportion of tasks that are of a routine nature as opposed to those requiring a high level of individual judgement. Drawn from a presentation given by Jane Clark (Sustainable Livelihoods Support Office, Department for International Development, DFID, UK) at the Maastricht workshop.

• Whose learning are we talking about?

At the three levels of learning outlined above, the development policy and programming focus mainly emphasises learning by donors and policy-makers. Hence, there is a risk that the participation of local actors is simply instrumental, i.e. more intent upon extractive actor consultation or 'rubber stamping' policies than effective participation in decision-making. The organisational learning approach recognises the complementary role of stakeholders and a multi-layered institutionalisation of findings. It emphasises the collective understanding of relevant development issues and finding the way forward. Finally, the societal learning approach recognises the existence of potentially insurmountable differences in perspectives among interested parties, yet at the same time acknowledges the need to negotiate sustainable answers to common challenges. In line with its focus on adaptive management and aspects of governance, it focuses on institutional development for improving societal learning and decision-making.

• Widespread learning versus the quality of the learning process

Unless evaluation instruments and evidence-gathering tools go hand in hand with an outspoken commitment to decentralising decision-making, engagement with local communities may increase the risk of co-opting them into existing development paradigms, thereby reinforcing rather than counteracting existing inequalities (Cooke and Kothari, 2001). At the same time, quality could be weakened by a lack of uniformity in the methods used, jeopardising the scientific 'rigour' of the evaluations. The counter-argument is that quality is enhanced by a greater richness of opinion. Unfortunately, to date, very few studies have paid much attention to assessing learning and its concrete outcomes.

• Power relationships and learning

Skewed power relationships, that are inherent to development practice due to the unequal access to resources, affect the process and outcomes of learning even when local ownership and mutual learning are intended. In their analysis of institutional learning with respect to local forest management, Engel et al. (2001) point to the need for the empowerment of local stakeholders and conclude that, even then, the national policy context may make or break the process at a local level. This leads us directly to the question of the management of learning: Who sets the rules? Who enforces them? Who evaluates the evaluators?

• Learning and the role of the evaluators

Where evaluation is used for learning and adaptive management, this implies a fundamental change in the role of the evaluator. From being a remote, research-oriented person trying to systematise the known and unearth the hidden, he or she becomes a process facilitator. His or her role is to help design and organise others' inquiry and learning effectively. Stakeholder analysis and communication skills, as well as the ability to manage group dynamics, become prime assets. Non-partisanship is a must, yet at times even objectivity may be challenged, e.g. when the empowerment of disadvantaged groups is felt to be a prerequisite for their full participation in joint learning. Enhancing learning will eventually mean a fundamental restructuring of the training, methodological baggage, professional skills and outlook of evaluators.

• Can learning help to mainstream the internalisation of evaluation results?

At first sight, pushing evaluation into a learning mode looks like an easy business. However, it leads more or less automatically to the next level: adaptive management, i.e. a systematic and continuous effort to adjust individual and organisational behaviour in accordance with the lessons of past experience.

Consequently, agencies that are prone to promote learning by others will be drawn irresistibly into asking questions about their own in-house learning capability. Hence, capacity development and the institutionalisation of internal learning processes emerge as the logical complements to systematic efforts to improve the internalisation of evaluation results.

• How can we create the conditions for effective learning?

Different approaches emphasise different enabling conditions for learning. The latter generally include the following:

• the absence of threats to openness and the sharing of opinions;
• curiosity and motivation on the part of the participants;
• the availability of intellectual and/or practical challenges, incentives and institutional support;
• opportunities for practical follow-up.

In addition, Guijt and Woodhill (2002) underscore the need for a systematic and explicit learning process and awareness of the minimum conditions that need to be met to make joint learning effective.

Box 3: Internalising evidence for social change: UNICEF and HIV/AIDS in Africa Currently, most HIV/AIDS communication strategies are based on one-way communication and behavioural change models. Their rationale is that the successful targeting and delivery of messages will cause individuals to change their behaviour. Such models, however, prove less successful when sexual relations or HIV/AIDS are at stake. Many social barriers, cultural practices and beliefs prevent messages from being followed up effectively. UNICEF has now developed a new approach to health communication in Eastern and Southern Africa. This consists of the following three steps: (1) Giving a voice to the voiceless and finding a way for the marginalised to express their realities. (2) Facilitating community conversations and devising action plans, with the participation of those who were formerly voiceless. (3) Building channels of communication with governments and donor agencies so that they can support the action plans developed by the communities themselves. Drawn from a presentation given by Neil Ford, RA Programme Communication UNICEF Regional Office, Nairobi, Kenya, at the Maastricht workshop.

Some lessons

'Let's look beyond the classical 'chicken and egg' analogy to scrutinise the rooster: Whose evidence are we talking about? How do we collect it and who owns it?' (Maastricht workshop participant)

'There is a real sense of urgency about the need for change. We have to transform our communication practices, quickly.' (Maastricht workshop participant)

'It's not just about cataloguing evidence. It's about connecting people and ideas and transforming information, via knowledge-sharing, into learning. However, new ways of working, new alliances, new champions also imply radical shifts in politics, power relationships and management performance issues.' (Maastricht workshop participant)

These conclusions may not be new, but they do underline the need to strengthen the effective uptake of evaluation results by means of learning.

• Evaluation policy

Improving the internalisation of evaluation results implies a shift in perspective with respect to the way in which development processes, and indeed development institutions, are managed. Adaptive management, i.e. flexibility and responsiveness to experience, must become a key institutional feature. As a consequence, institutional development is a necessary complement to learning by development agencies.

The balance between accountability and transparency on the one hand, and learning and adaptive management on the other, needs further scrutiny. At the same time, it is important to be aware of any power imbalances between donors and aid recipients. We shall need to develop ways and means of safeguarding the credibility of evaluators and development professionals engaged in learning through evaluation.

• Evaluation practice

In terms of methodologies, what is needed is a pragmatic, process-oriented approach rather than a dogmatic, content-oriented one. There is no single set of tools that guarantees learning. Indeed, many different ones can be used if the principles governing successful learning are taken into account. These include mutual respect, inclusive thinking, a readiness to understand other peoples' perspectives, and a willingness to take criticism seriously and to revise institutional cultures, rules and procedures.

Complexity, unpredictability and uncertainty are regular features of multi-actor learning processes. Methodological issues 'extend far beyond simply which method works best, as these are just a small part of the extensive communication processes that lie at the heart of monitoring and evaluation.' (Guijt, 2000)

Box 4: Internalising evidence for social change: The Australian Land Care programme The Australian Landcare programme emerged because of growing concerns about salinity, soil erosion, water quality and biodiversity decline in rural areas. A community-led network emerged of more than 4000 Landcare groups involving farmers, scientists, politicians, business and community leaders in gathering and debating evidence aiming at overcoming land degradation and improving agricultural sustainability. Its impact was visible in the high level of community mobilisation and an increased societal understanding of the problems. Various participatory forums/platforms were established at local, regional and national level. Conditions that helped trigger effective social learning were: (1) appropriate forums and platforms, (2) clear mandates and legitimacy for the forums and the process, (3) engagement of all key stakeholders, (4) incentives for participation, (5) integration with existing institutions and processes, (6) a clear scope and boundaries for learning and change, (7) coordination at and between different levels, (8) effective facilitation and leadership, (9) utilisation of a diversity of methodologies, (10) effective establishment and monitoring of performance questions and indicators, (11) consciousness of what social learning is and why it is needed and (12) the questioning of overarching policy-paradigms. Drawn from a presentation given by Jim Woodhill, IAC, University of Wageningen, the Netherlands, at the Maastricht workshop.

• Evaluation professionals

The need to rethink the professional profile of evaluators is paramount if they are to facilitate learning. Apart from having excellent research and communication skills, the evaluators of the future will also need to possess process facilitation, conflict resolution and negotiating skills. Analytical skills will have to include tools for situation and stakeholder analysis, as well as for understanding conflict and group dynamics.

Box 5: The challenges ahead: the shifts involved in strengthening learning through evaluation In evaluation practice: From advice on issues to the facilitation of reflective process From consulting to valuing local actors, evidence and knowledge From single expert to nested, multiple learning From output assessment to the management of joint inquiry From theoretically constructed to 'harvested' performance indicators From talking to asking and listening In institutional management and culture: From adapted to adaptive management From an expert culture to a learning culture From practices locked in structures to institutional learning In evaluation theory: From input/output logic to recognising 'fuzzy' realities and emergent outcomes From a linear understanding to dealing with unpredictability and cascades of feed-back From aiming at short-term simplicity to dealing with long-term complexity From theories of planning to methodologies of connecting In evaluation policy: From predefined outputs to open-ended learning From evaluating ambitions to understanding outcomes Based on concluding remarks made by Paul Engel at the Maastricht workshop.

• Evaluation research

Far more emphasis needs to be placed on developing effective conceptual and practical tools. So far, research has hardly touched upon the systematic assessment of the learning effects of different monitoring and evaluation approaches, tools and techniques.

• South-North and South-South exchange

The flow of lessons on practices and tools from the South to the North, and between Southern actors, is a field of inquiry in which much can be learned from systematic collaboration among evaluation researchers and practitioners.

References

Cooke, B., Kothari, U., (2001). Participation: The New Tyranny?, Zed Books, London.

Engel, P.G.H. and Salomon, M. (1997). Facilitating innovation for development: a RAAKS resource box, KIT Press, Amsterdam.

Engel, P.G.H., Hoeberichts, A. & Umans, L. (2001). 'Accommodating multiple interests in local forest management: a focus on facilitation, actors and practices'.
In: International Journal of Agricultural Resources, Governance and Ecology, Vol.1.

Guijt, I. 'Methodological Issues in Participatory Monitoring and Evaluation' (2000). In: Blauert, J., Campilan, D., Gaventa, J., Gonsalves, J., Guijt, I., Johnson, D., Ricafort, R. (eds.) Learning from Change: Issues and Experiences in Participatory Monitoring and Evaluation. Intermediate Technology Publications, London.

Guijt, I. and Woodhill, J. (2002), with Berdegué, J. and Visser, I. Learning through
E-networks and Related M&E Issues. FIDAMERICA/Grupo Chorlaví.

Holling, C.S. (1995). What Barriers? What Bridges? In: L.H. Gundersen, C.S. Holling, and S.S. Light (eds.) Barriers and Bridges to the Renewal of Ecosystems and Institutions, Colombia University Press, New York.

King, C. and Jiggins, J. (2002). 'A systemic model and theory for facilitating social learning'. In: Wheelbarrows Full of Frogs: Social Learning in Rural Resource Management. Leeuwis, C., and Pyburn, R. (eds.), Van Gorcum, Assen, the Netherlands.

OECD/DAC. (1997). Evaluation of Programs Promoting Participatory Development and Good Governance. Synthesis Report. OECD/DAC Expert Group on Aid Evaluation.

OECD/DAC. (2001). Evaluation Feedback for Effective Learning and Accountability.

Röling, N.R. (2002). 'Beyond the aggregation of individual preferences: moving from multiple to distributed cognition in resource dilemmas'. In: Wheelbarrows Full of Frogs: Social Learning in Rural Resource Management. Leeuwis, C., and Pyburn, R. (eds.), Van Gorcum, Assen, the Netherlands.

Sutton, R., (1999), The Policy Process: An Overview. ODI Working Paper 118. London.

Uphoff, N., Combs, J. (2001). Some Things Can't be True But Are: Rice, Rickets and What Else? Unlearning Conventional Wisdoms to Remove Paradigm Blockages, Cornell International Institute for Food, Agriculture and Development, Cornell University, New York.

Van der Veen, R.G.W. (2000). 'Learning Natural Resource Management' In: Guijt,
I., Berdegué, J.A. and Loevinsohn, M. (eds.) Deepening the Basis of Rural Resource Management. ISNAR & RIMISP, The Hague/Santiago de Chile.


1 For the purpose of this brief, we use the term evaluation in its broadest sense, encompassing the gathering of evidence as well as its processing and valuing. It also includes the more extensive societal interaction with respect to evidence gained through monitoring and evaluation.