Review of 2000 - 2004 Country Portfolio Review Reports

Evaluation Team

This report was prepared by a team comprising Messrs. M. Manai, Chief Evaluation Officer, A. Coker, Principal Evaluation Officer, W.Byaruhanga (Consultant) under the supervision of Getinet W. Giorgis, Director of OPEV. Ratings were made and peer review provided by Mr. E. N’diaye, Chief Evaluation Officer, Mrs. G. Yirga-Hall, Messrs. P-A Rochon, F. Turay, and H. Razafindramanana, Principal Evaluation Officers, all of OPEV.

Objective

The principal objectives and scope of CPRs, as cited in the policy document (Ref. ADF/BD/WP/95/01/Rev. 1), were to:  • Improve the quality of the Bank’s country portfolios;  • Assist member countries in meeting their current obligations;  • Adapt projects to changing economic circumstances; and  • Provide guidance for future country programming.    This CPR review assesses the process and content of CPRs undertaken during 2000-2004. Recommendations are made to strengthen portfolio performance and increase   the development impact of projects. The review is to set the agenda for future CPR development.

Main Recommendations

Recommendation(s) to the Bank:

• Policy guidelines and format: a. Update the current policy, guidelines and format to incorporate changes in emphasis and approach. B. Train staff in performance rating and the contents of new guidelines"

• Content: a Focus less on project-specific issues addressed by supervision missions, and more on systemic and generic issues related to key portfolio management issues. b. Pay more attention to the assessment of links between physical implementation, CSPs and expected development outcomes. c. Reduce background discussion and instead introduce discussion on progress made since the last CPR. d. Harmonize the rating scales and methodology for supervision reports, project completion reports and performance evaluation reports.

• Processing of CPRs: a. Increase the frequency of CPRs; at the very least respect current guidelines. b. Give country directors a greater say in the composition and duration of missions. c. Each fiscal year, after consultation with borrowers, country directors should select countries for CPRs based on portfolio size and/or portfolio problems. d. Prepare a discussion note or Issues Paper and forward it to borrowing countries for their approval well in advance of the CPR mission. e. Raise the profile of CPRs, both in the ADB and the government. f. Create a mechanism to follow up and monitor action plans that have been agreed on, and are owned by client countries.

• Common and systemic problems: a. Mainstream M&E capacity building in its interventions in regional member countries. b. Strengthen accountabilities and incentives, especially for sector managers, to enhance the candor and quality of supervision and the results focus of project performance ratings. c. Reduce delays in effectiveness; meet key conditionalities before Board presentation as far as possible; assist in preparing bidding documents; and clarify to countries the conditions or client commitments that are preconditions to loan effectiveness. d. Improve the quality of supervision through a better skill mix and greater intensity of missions and greater use of peer reviews. e. Equip Bank country offices in regional member countries with an adequate complement of experienced staff to assist in improving portfolio performance.

• The 1995 policy guidelines have been useful in guiding the CPR preparation process and periodically assessing the IP of the Bank Group portfolio. Since then, there has been various developments in the operational policies of the Bank. First, there is an increased awareness of countries’ need for greater ownership of, and responsibility for their portfolio performance monitoring. Second, there has been a strong movement for closer attention to links between project implementation (process and outputs), expected development outcomes and sustainability of benefits. Third, Bank management has lately been acting on various fronts in implementing the results agenda, including the results reporting system. These developments call for the need to update the policy, format and guidelines of the CPR process. 

• At the policy level, management may want to consider two basic principles identified as good practice: (a) the CPR has to be useful to the country for managing their projects and resolving problems, as well as being an integral part of the dialogue between the Bank and the national authorities; and (b) the CPR has to be useful to the Bank in tracking and monitoring inputs, outputs and intermediate outcome indicators at country level. The CPR also has to provide advance information on the level of operational compliance to Bank rules, procedures and requirements, the assessment of related risks, including fiduciary risks and safeguard risks, and the increase in development effectiveness. 

• Under the updated policy guidelines for CPR preparation, management may want to consider a CPR that is focused on the track record of Bank-assisted projects and operations in achieving intended results on the ground. More weight should be given to key systemic and generic implementation issues. These include the direction of future assistance and implications for portfolio management, with an action plan and timetable for resolving identified issues. 

• The CPR, while covering the central issue of project performance, should also review broader issues pertinent to the country portfolio. This should be done from the perspective of the key sectors and strategic thematic issues covered by the portfolio. The major challenges ahead should be identified and the key trans-sectoral issues emerging from the portfolio review examined

• The CPR process should be transformed into a more forward-looking course of action. It should be prepared in close collaboration with the borrower, project coordinators, key stakeholders and other donors. This approach will promote awareness among all concerned of common problems affecting portfolio performance. It will also help to secure borrower ownership of, and commitment to Bank Groupfunded operations in the portfolio.

• There is a need to strengthen the link between CPRs and CSPs. Comprehensively prepared CPRs, which should include analyses of lessons learned, strategies and priorities, should precede the preparation of CSP and CSP updates. This will guide the CSP in delineating priority sectors for the strategy for future Bank Group assistance

• Management may want to consider emphasizing project-specific issues less and focusing instead on higher systemic and strategic issues. It should provide for sector studies, sector workshops and seminars (and the use of their results), to promote a more informed understanding of sector policies, strategies and priorities. The policy should allow the CPR report to draw from the findings in the sector reviews, and include a discussion of recent developments in relevant sectors.

• The new format should follow the updated approach and objective of CPR processes. In particular, it should provide for additional areas of analysis and coverage. For example, progress made since the last CPR should be routinely reported and be made key topics of the development impact of projects, the institutional development impact and the discussion on issues affecting the sustainability of projects.

• The processes and systems should be improved to improve the frequency and coverage of CPRs and raise the profile level of CPR missions. Country directors should be given control of the resources for CPRs, or empowered by other means to take full charge of the CPR process.

• Mandatory leadership of missions by directors or, at the very least, country managers, should be considered. The vice-president’s permission should be sought where lowerlevel staff has to lead the mission

• Responsibility for oversight of portfolio quality should expressly be assigned to directors of country departments. The role of country offices in portfolio monitoring and reporting should be clarified in explicit instructions

• Portfolio performance rating and reporting methodologies should be harmonized and consistently applied. Operations staff and managers should be trained in the new context of results-based management.

• CPRs are currently narrowly focused on cataloguing individual projects and their problems. A CPR should review broader issues pertinent to the country portfolio from the perspective of key sectoral and thematic areas, and identify the major challenges ahead. It should also examine significant intersectoral and crosscutting issues emerging from the portfolio review. 

• In addition to the requirements of the existing format, future CPRs should be more analytical. They should include a discussion on the achievement of DOs, portfolio sustainability, institutional development impact, borrower performance and Bank Group performance. Future CPRs will need to provide answers to a set of evaluation questions that the current CPRs are not addressing adequately. These are: In what ways can the M&E and supervision system be improved by using the CPR? In what ways can the CPR ensure that project goals are consistent with the sector and country strategies? How can the CPR process contribute to the achievements of DOs strategy? In what ways can the review enhance the ownership and commitment of client countries? How can the Bank and the borrower ensure the sustainability of projects in the portfolio?

• The assessment of Bank Group performance should include an analysis of service quality, especially of those tasks for which the Bank has primary responsibility. For instance, has the Bank Group provided appropriate advice throughout the project cycle? Was it flexible in adapting to changing circumstances? Did it oversee project activities satisfactorily? The analysis should also focus on the Bank Group’s role in ensuring project quality at entry, satisfactory implementation and future operation. 

• CPR content should provide detailed and candid assessments of rating criteria. Project staff and managers should be trained and guided in the methodology and extent of coverage in narrative assessments. Managers should establish incentives and accountability for more reliable, timely risk and progress monitoring ratings, and review them adequately before signing them off. 

• The POPR and OPEV should accelerate the harmonization of rating scales and methodology for reports on supervision, project completion and performance evaluation. 

• The review has identified several portfolio performance indicators on which action is needed. Delays in project effectiveness and execution of projects remains to be a problem. The quality of supervision and weak monitoring of the portfolio at the country level is becoming a concern. Risks and factors that most influence project outcomes are not identified and tracked systematically. In this connection, the proposed review of Bank Group Supervision Monitoring System that CODE has instructed OPEV to undertake is timely. 

• The frequency of supervision missions needs to be increased, as well as the number and skill mix of staff days, particularly for projects with weak executing agencies. If missions are intensified and their duration enhanced, project staff would become more involved. They could assist or advise project executing agencies or implementation units on implementation activities. Supervision missions should be expanded to include the Bank’s support departments, in particular the procurement unit and disbursement division. 

• It is necessary to pursue the recommendations of CPR and supervision reports. Concerted action is needed, especially as regards the action plans for resolving generic problems that hamper project implementation, such as compliance to Bank rules and procedures and reporting requirements (progress and audit reports). Early discussions with governments should be held to establish realistic conditions and time schedules. Conditions of loan effectiveness should be appropriately timed and kept to a minimum. 

• The recent move to establish country offices in several countries will be helpful. However, the offices should be provided with an adequate complement of experienced staff. These offices should be able to play an active role in: (a) the physical and financial monitoring of projects in the portfolio; (b) the monitoring of disbursement requests; (c) payment of counterpart funds; (d) regular preparation of quarterly progress and audit reports; (e) country-level coordination with other donors; and (f) improve communication between the borrower and the Bank.