This summary report was prepared by Guy Blaise Nkamleu, Principal Evaluation Officer, under the supervision of Odile Keller, Manager, and Mohamed Manai, Manager, with guidance from Rakesh Nangia, Director of the Operations Evaluation Department (OPEV) of the African Development Bank. The work also benefitted from comments and discussions on substantive issues with colleagues across the Bank at its different phases. 

Objective

The purpose of this review is to assess the Bank’s ESW over the period 2005-2010 and to draw lessons and formulate recommendations to improve the effectiveness of future ESW. 

This study should be seen as a formative evaluation,as the Bank is putting in place an internal process for ESW.  

A theory of change behind ESW has seven stages: 

1. Developing a vision for ESW (policies andstrategies). 
2. Production. 
3. Processing and packaging. 
4. Dissemination. 
5. Receipt by stakeholders. 
6. Stakeholder engagement with messages. 
7. Stakeholders take action based on messages.

Main Findings

• The Bank has never had a specific and comprehensive policy for ESW. Although not always explicitly mentioned, elements of the ESW policy and strategy are found in the Bank’s Operations Manual, Organization Manual, and the Bank’s knowledge strategies. The findings point to a need to consolidate, harmonize and make them explicit

• The volume of ESW prepared has increased substantially since 2008, with increased focus on the Bank’s strategic areas. In response to the growing demand for cutting-edge knowledge, international development organizations publish studies and report to both enhance the quality of lending and as a business line for policy and program advice for clients etc

• The ESW portfolio remains dispersed, difficult to find, and inconsistent. The Bank has no corporate definition of ESW and keeps no systematic records, leading to confusion over what constitutes ESW. The Bank produces ESW in all areas, with varying orientations and depth. In some cases there is duplication with other donors who have already prepared similar ESW reports

• In recent years corporate planning and programming of ESW and other non-financial activities have improved through the threeyear rolling program and budget. However, these still do not provide a precise overview of the Bank’s activities. The Bank consistently fails to realize planned ESW while initiating unplanned ESW largely in response to resource availability rather than to strategic needs or actual demand based on operational needs. This is compounded by poor coordination and prioritization across and within Bank units

• Lack of systems. There are no mechanisms for managing ESW as a portfolio, no comprehensive database with terms of reference and reports, and no harmonized procedures for quality control and/or quality assurance. Poor coordination across Bank units and between field offices and headquarters in the same unit is a handicap to effectively managing the Bank’s ESW.

• Dissemination of the Bank’s ESW is limited, and the ESW portfolio has little visibility in Regional Member Countries (RMCs). The Bank’s field offices do not have a clear role in disseminating ESW. Generally, there is very little attention to modalities and budgeting for dissemination. Currently, dissemination is ad-hoc, with no formal guidelines to help choose appropriate dissemination strategies. Posting on the Document and Records Management System (DARMS) and presenting to the Board are considered to be the end stages for many ESW documents, but many are neither presented to the board nor posted on the web

• Use of ESW products - by the Bank itself, by RMCs, and by the public at large - is low. When ESW is used, it tends to be undermined as the Bank down-plays its role in influencing decision-making. The Bank tends to be less willing to using ESW to push the envelope on policy dialogue with member governments even when ESW could present a sound case for policy change. This raises vital questions related to the overall purpose and objectives of the ESW for the Bank.

Main Recommendations

Recommendation(s) to the Bank:

• The Bank needs to contextualize ESW, and provide a clear definition. This entails at least two aspects: (i) deciding whether the Bank should continue using the term “ESW” and what it means for the Bank; and (ii) clearly position ESW within the Bank’s knowledge ecosystem. (i) The first aspect entails at least three options: 1. Take the term and its baggage. The Bank could continue using “ESW” with the definition used by the World Bank, who originated the term. This has the advantage of harmonizing the meaning of ESW between the two institutions. 2. Take the term and leave the baggage. Over time other MDBs have used different definitions for ESW, depending on its role and function. The World Bank definition does not necessarily align with the definition used by other MDBs or AfDB’s ambitions for this product. In this scenario, the Bank could create its own definition of ESW to reflect the orientation it seeks to give to its ESW within the umbrella of knowledge products. 3. Leave the term and its baggage. The World Bank clearly intends countries to be the end-users of ESW and uses ESW to push for policy change. The World Bank has greater technical capacity and more willingness to challenge government policy positions. It is also more prone to question official statistics and analyses. ESW, as defined by the World Bank, is a good instrument to help fulfill its own objectives. The AfDB needs to clarify its overall objectives and ambitions for this instrument as it defines ESW. The AfDB would still need studies to inform its lending but would not necessarily need to call them ESW. A term such as “country analytic work” with the definition used by the OECD may be an option to consider. However, this does create issues for regional and cross-country products. (ii) On the ESW positioning, much confusion exists among ESW, research, and internal reports (such as portfolio analysis), with unclear audiences for Bank ESW. The Bank should revisit its knowledge strategy to clarify the function, purpose, and audience of its analytical and knowledge products. Adapting from the “general framework for knowledge in organizations”, one option is to organize Bank knowledge products into three main groups: 1. Knowledge for external clients is work done for specific countries and at the regional or continental level. This category includes ESW. 2. Knowledge as a public good is work that has no pre-specified client but that ultimately advances knowledge in specific area of development. This category includes products from the Chief Economist Complex (African Development Report, research publications to stimulate debate etc.) 3. Knowledge for internal use is analytical work intended to inform Bank staff on the latest approaches or findings and new instruments or products for clients. This category includes sector reports, policy papers, and portfolio investigations and tools developed primarily for Bank staff. 

• Adequate administrative budget should be set aside to finance strategic ESW, with CSPs defining the envelope to be allocated to ESW. Trust Funds should be used only for unplanned, additional or non-core ESW. The CSP should define the related core ESW that needs to be conducted and the policy dialogue grounds on tangible ESW. CSP teams should split the proposed ESW into two groups: i) strategic ESW funded by the administrative budget; and ii) ESW linked to trust fund availability. Another option to consider is to establish a dedicated trust-fund similar to the model of the Knowledge Management Trust Funds being proposed by the Chief Economist. 

• ESW should be more carefully aligned with Bank strategy, and demand-driven ESW should be a priority. The Bank’s ESW is largely supply-driven, even when governments are the end-users. The Bank overemphasizes country needs for ESW, and few government officials actually read such reports. Studies specifically requested by clients are more likely to find an audience among policymakers. A serious needs assessment should be done before planning for ESW, and these should be more carefully aligned with the Bank’s LTS and country strategies. In addition, annual programming of ESW should take place at least at the complex level to ensure flexibility and capacity to respond to demand. 

• The Bank needs to clarify the role of the Field Offices and Regional Resource Centers with respect to ESW. The Bank’s rapid decentralization is bringing about more decision centers and Bank rules need to be responsive to these changes. The lack of guidelines on managing and disseminating ESW provides no clearly articulated mandate or expectation regarding the role of field offices in generating, managing and promoting ESW. 

• The Bank should create a business process and increase corporate oversight to support quality assurance, dissemination and management of ESW as a coherent portfolio across the Bank. There is no clear responsibility for ESW between country and sector departments. Most sector departments produce ESW in “silos” which risks loss of a regional or country perspective. The Bank should strengthen connectivity between departments in producing and disseminating ESW. Regional departments need to be the focal points for coordinating all ESW for their region. This should ensure that ESW is in line with country and regional strategies. For quality assurance, one option is to subject ESW to a review process similar to that used for lending operations and CSPs, including reviews by external experts. Another option is to have a good arrangement for support from the Chief Economist’s Complex which would be responsible to ensure minimum quality. In any case, the Bank needs to task a Bank unit (such as the strategy department or ORVP) or establish a knowledge strategy group to oversee the Bank’s ESW portfolio. Even well targeted and relevant ESW needs to be disseminated to be useful. Instead of posting ESW on the web and expecting end-users to find and make use of it, the Bank should explore and utilize different avenues for dissemination. This will require incentives and a systematic budgeting for dissemination of all ESW 

• The Bank needs to be realistic with respect to its actual capacity to deliver ESW. Key performance indicator targets for ESW are established without considering the Bank’s actual capacity. The Bank ought to target a smaller number of high-return ESW. These should be for selected topics, partners, and pursue institutional arrangements on division of labor with other agencies and organizations.