Evaluation Team

This summary report was prepared by Albert-Eneas Gakusi, Chief Evaluation Officer with the guidance of Odile Keller, Manager of the High Level Evaluations Division at the Operations Evaluation Department  (OPEV) of the African Development Bank. Franck Perrault, Acting Director of OPEV (May 2011- January 2012) and Rakesh Nangia, Director of OPEV from January 2012, provided overall guidance. The evaluation also benefitted from the support and comments from Mohamed Manai, Manager of the Project and Programme Evaluations Division.

Wrap Up video of the evaluation from the task team leader:Albert-Enéas Gakusi, Chief Evaluation Officer

Objective

This evaluation assesses: 

(a) the relevance and consistency of the Bank’s strategic and operational framework for fostering regional integration; and the

(b) relevance, efficiency, effectiveness and sustainability of the Bank’s multinational operations (Mos: operations taking place in at least two countries). It discusses the performance of the Bank and key factors in that performance, the Bank’s organisational effectiveness, and makes recommendations for improvement. The evaluation covers the period 2000-2010. The evaluation focuses on Mos and does not include single-country operations that contributed to regional integration. These are not captured in the Bank’s data systems. As a  consequence, the evaluation is unable to assess the Bank’s overall contribution to regional integration.

Main Findings

• The Bank has developed a sound strategy for its activities in relation to regional integration, as well as a programming framework to improve the strategic focus and ensure greater selectivity in the use of the ADF resources. However, further improvements are required

• The Bank has significantly increased its share of MOs from 6 % (2000) to 15.4 % (2010) of total approvals. Sector distribution of the MO portfolio has evolved in line with the priorities of the Mid-Term Strategy and ADF. The Bank has responded to the growing demand for MOs by increasing the ADF resource envelop dedicated to regional integration, while setting up strong financial incentives for ADF countries.

• MOs achieve their objectives no less effectively than single-country operations. There is no evidence that the Bank’s MOs are less effective, less efficient  or less sustainable than single-country operations even though they are more complex and face more difficulties at the design and implementation stages. Possible explanations for the better performance of Mos range from imperfections in measurement and reporting tools, to the hypothesis that Mos which have gone through the relatively rigorous selection and appraisal processes are particularly fit to withstand adverse conditions. Sustainability and efficiency remain a challenge for most Mos as well as for single-country operations..

•  Key factors in the performance of MOs include country commitment and ownership, implementation and governance arrangements, and a conducive policy environment. A QAE review of Mos approved between 2006 and 2010 found that the Bank has performed “moderately well or better” in those areas. However, increased alignment on national and regional priorities has not been matched with strong analysis of alternatives, adequate cost and benefits calculations (a particular issue for non-infrastructure projects, although even infrastructure projects lack cost-benefit analysis, especially at the project completion stage), and risk assessment to allow governments to make informed decisions on the opportunity costs of engaging in MOs. The Bank has a strong record of describing in detail the implementation and governance arrangements (capacity of the implementing agency, roles and responsibilities, etc.), but seems to give more limited consideration to the underpinnings of the specific governance arrangements. Finally, the contribution of operations to development outcomes is more likely to be sustainable if accompanying measures and policy reforms are adopted at country level. Although the regional integration agenda is being increasingly integrated into CSPs, field  offices have not yet engaged in the kind of strategic policy dialogue required to ensure that the necessary conditions and political commitment are in place for sustaining project outcomes.

• The Bank’s capacity to implement its regional integration agenda has significantly improved with the creation of ONRI, although further specification of its mandate and clarification of responsibilities of the different departments involved in regional integration could improve its effectiveness. ONRI was a turning point in efforts to strengthen the Bank’s strategic vision and improve its capacity to deal more selectively with MOs. However, reflecting the lack of focus of its strategic framework and mandate, ONRI is yet to develop a focussed program of work on the soft dimension of regional integration. ONRI’s relations with the Bank’s other departments are crucial for promoting a regional integration approach across the Bank, but its role in relation to policy and strategy formulation versus the role of these departments is unclear. There is limited evidence that lessons from monitoring and evaluation influence the programming or design of MOs, particularly since there is no department tasked with compiling and disseminating those lessons

• The Bank’s single-country operational model is not adapted to the specific requirements of MOs. For example, staff incentives do not account for the complexity of MOs. Existing tools (ratings, format of PARs, dissemination of information) are imperfectly adapted to the specificity of MOs, and do not always help to ensure transparency, accountability and learning. Field offices do not have clear responsibilities in relation to MOs, and the policy dialogue at country level does not include sufficient consideration of the regional integration agenda

 

Main Lessons

• If the benefits of a multinational road are to fully realised, it is of paramount importance to solve the problem of the OBSP at the design stage of a project. It is also necessary to overcome nonphysical barriers, including harmonisation of the of policies, rules, standards and procedures national regulatory authorities; as well as road blocks and border controls, which are seen as concrete limits to trade.

• A rigorous methodology for cost and benefit sharing is key for achieving an equitable distribution of project outcomes and ensuring the long-term consensus and commitment of participating countries. 

•  To enhance the development impact of regional infrastructure projects, complementary national policies need to be designed and opportunely implemented 

• The absence of independent regional regulatory authorities to enforce the agreed contract rules can be an obstacle to effectiveness in case of disagreement between operators. 

• Donor coordination needs to go beyond cofinancing agreements and the setting up of project implementation committees, to consider the soft and policy measures that complement and sustain the results of infrastructure projects. 

•  The strong convergence of development priorities and the objectives of public and private stakeholders of the MO, along with adequate return to investment, are key to successful implementation and sustainable results. 

• The provision of basic services (schools, clinics, boreholes, training, etc.) for communities impacted by large productive infrastructure is of paramount importance for ensuring the sustainability of the operation. 

Main Recommendations

Recommendation(s) to the Bank:

• The Bank should develop a clear, institutionwide definition of MOs as operations that contribute to regional integration. This definition should be based on a set of criteria related to the results chain of the strategic framework. Those criteria should be applied to both private and public sector operations; and should also be used to identify single-country operations contributing to regional integration. The Bank’s information systems (in particular SAP) should be adapted to distinguish operations along two separate dimensions: (a) those taking place in one or several countries; and (b) those that make (or do not make) a significant contribution to regional integration. These categories would help improve reporting on the Bank’s role in regional integration, and enable the Bank to better capture its overall contribution. 

• The Bank should be more focussed when addressing the soft constraints of regional integration, and specify the areas for providing RGPs. For soft constraints, one option would be to concentrate on the constraints of the regulatory and administrative framework in relation to the sectors where the Bank is most active. For RPGs, the Bank should define a limited number of areas where it has appropriate expertise and where it can contribute the most compared to other donors. 

• The Bank should define the role of private sector operations, taking into account the contribution that such operations - and the private sector more generally - can bring to fostering regional integration. 

• A mechanism for systematic feedback and learning from experience should be established to influence the design of new MOs, especially in relation to the key factors of performance. This mechanism should specify clear responsibilities for the collection, validation, analysis and use of the information for policy and program formulation. It should encourage internal learning on the specific characteristics of Mos (e.g., governance agreements, legal arrangements underpinning these arrangements, cost sharing), and – given the time required for the implementation of such operations – ensure that learning takes place not only at completion but also during implementation. 

• The mandate and resources of ONRI should be aligned. The Bank should expand ONRI’s resources to fit its mandate, or focus ONRI’s mandate to fit existing resources. ONRI’s mandate should consist in creating maximum value for the Bank rather than responding to ad-hoc demands. It should focus on providing high-level and strategic advisory services, knowledge generation and management, and deeper engagement with RECs and other regional institutions in Africa. 

• The Bank should clearly define the roles, responsibilities and division of labour among ONRI, regional departments and sector departments. Regional and sector departments should designate focal points to engage with ONRI and have responsibility for mainstreaming regional integration into CSPs, sector strategy and operations. Responsibilities for monitoring and reporting on RISPs should be specified. Field offices’ responsibilities should be better defined and their capacity strengthened to engage on strategic policy dialogue on regional integration issues. These roles and responsibility should be clearly specified and disseminated across the Bank. 

• The Bank’s tools and business model should be adapted to the specificities of MOs. Necessary measures include, but are not limited to: (a) defining a set of specific criteria for the MO readiness review; (b) assigning overall responsibility for the MO to one task manager; (c) allocating more time and resources for the design and supervision of MOs; (d) reconsidering the format of PARs for MOs; and (e) adapting the incentives for staff to engage in complex, crosssectoral Mos.