Evaluation Team

This report was prepared by Detlev Puetz, Principal Evaluation Officer (OPEV), with contributions from James Keough (Consultant) and Athan Coker (former OPEV Principal Evaluation Officer). Extensive research assistance was provided by Diomandé Mabarakissa. The finalization of the report was supervised by Colin Kirk, Director, OPEV. Douglas Barnett (former Acting Director, OPEV), Getinet Giorgis (former Director, OPEV), Afework Aklilu (former Principal Evaluation Officer, OPEV), and he evaluation team of the Inter-American Development Bank provided early inputs into the report. Victoria Elliot and other peer reviewers provided valuable suggestions on focus and presentation. All questions and comments should be referred to Mr. Puetz at d.puetz@afdb.org.

Objective

This evaluation comes at a point when Management is re-considering and redesigning the project supervision system and operations manual at the Bank. The evaluation is expected to add value to this effort to become more results-oriented by providing key evidence on supervision performance, reasons for this performance, and best practices. Based on extensive data reviews and staff interviews (Box 1) the evaluation looked at three questions:

(1) Are we doing the right things in supervision? (Is supervision relevant?)

(2) Are we doing what we said we would do? Are we doing things right? And to good quality standards? (How is performance?) and

(3) Do we know how we are doing? How do we manage our information? How do we report and communicate? (Managing information)

Main Findings

• Much of what needs to be done in project supervision at the Bank is well-known and straightforward, but it needs to be done fully and effectively. Bank Management already recognized this and included a number of key performance indicators on supervision and implementation in the Bank’s institutional targets.

• The evaluation finds that supervision policies in the past were by and large relevant and were for the most part clearly defined. But results and risks were insufficiently addressed in supervision policies and practice. Also, institutional responsibilities in supervision were not sufficiently established within a multi-disciplinary team approach. Supervision guidance and oversight suffered from the relevant department’s low capacity to implement and deliver on its mandate and poor Bank wide management information systems. The evaluation concludes that the current policies and institutional arrangements of supervision are not fully relevant and geared towards the Bank’s future needs. The paradigm of project supervision at the Bank is not up-to-date with emerging opportunities and needs, particularly those of decentralization, results and risk orientation, and partnerships. The efforts by management to come up with a new policy, shared responsibilities, and new procedures for supervision are therefore timely and, given the importance of supervision in achieving the Bank’s goals, should be given priority.

• Performance of supervision has been marginally improving over the past years, yet remains overall unsatisfactory. Operations are field supervised on average about 1.5 times per year. Yet, only half of all projects are supervised 1.5 times per year as required by policy.One in five projects is not field supervised at all in any given year which is not acceptable. Further, the Bank does not systematically differentiate supervision by project type and risk category. While frequency of supervision missions matters, quality is equally important. Quality standards are currently not well defined, monitored, and enforced. Supervision instruments beyond periodic supervision missions are underperforming, in particular launching missions, mid-term reviews, and external audits. Their contribution to an integrated, results-oriented supervision system could be substantially improved. Although efforts have been made by management to enhance performance, supervision appears to be under-resourced in view of the importance of its role in ensuring that operations are efficient and effective. Transparency with regard to the resources going into supervision is lacking. Potential savings through decentralization and smarter project design have not yet been well exploited.

• Project risks are not well managed at the Bank. This affects performance. Two out of three Bank staff believe that the identification of projects at risk is ineffective. They point to a number of deficiencies in the system for rating project performance and in follow-up on supervision recommendations. The Bank’s electronic SAP data and management information system for projects is highly unsatisfactory and “archaic”, despite some improvements such as the data warehouse system. The availability of up-to-date and reliable information on projects at all levels of management remains unacceptably low. Transparency in project management and supervision is limited. There is no central project information hub with relevant project documents for reference.

• In conclusion, it is clear that the Bank’s supervision system, despite recent improvements, needs to be overhauled to deliver the gains which an effective supervision system could provide in terms of operational effectiveness, efficient and impact. The staff survey indicates that operations staff are impatient for such a system which would remove many constraints and allow them to achieve better results. The evaluation therefore concludes with a set of recommendations requiring changes to be made across the Bank

Main Recommendations

Recommendation(s) to the Bank:

• The Bank should pay more attention to results and risks in supervision policies and institutional arrangements, particularly in terms of integrating these aspects into operational guidelines and everyday supervision practice. This should particularly include the development of a comprehensive concept of project risk management at the Bank. An efficient and effective results-based supervision system must be first and foremost risk-based. Secondly, consistent efforts are required, particularly at country level, so that project-level monitoring and evaluation systems function more effectively to ensure the performance of the broader results system .

• The Bank must develop an appropriately resourced project supervision system with better integration supervision instruments and activities, while ensuring differentiation by project type and phase. This entails an expansion of team work in supervision, more decentralization to the field, and better integration of the various supervision instruments. The frequency and quality of supervision instruments beyond regular periodic missions must be enhanced, particularly of launching missions, mid-term reviews, and external audit reports. Supervision policy and practice should be better brought in line with the requirements of different project categories and project cycle phases; and in proportion to project size and risk exposure. Supervision guidance and oversight need more attention. The Bank must determine an appropriate resource envelope for supervision, in line with policy requirements, and needs to better monitor the resources expended on supervision on the basis of full-cost accounting.

• The Bank needs to pay more attention to the quality of supervision and the qualityb of follow-up on supervision recommendations. The quality of activities and inputs into the supervision process needs to be enhanced, such as the length of missions, appropriate skills mix during missions, project ratings, and staff incentives. Equal attention needs to be paid to the quality and communication of supervision outputs, in particular supervision reports and follow-up on supervision recommendations. The quality of supervision must be measurable, with clear indicators and targets that can be regularly monitored.

• The Bank urgently needs to overhaul and upgrade the SAP electronic project data and management information system for projects. The SAP projects system requires urgent redesign to ensure better quality, reliability, and scope of supervision and other data; to enhance the interface and user-friendliness of the system; and to generate transparency and accessibility of important project data and documents. The SAP projects system, in conjunction with the Data Warehouse System, should be developed into a central hub to serve the various project and portfolio data needs and management functions at the Bank, particularly to support follow-up of supervision recommendations.