Evaluation Team

The evaluation was commissioned by the Bank’s Operations Evaluation Department (OPEV) and carried out by Professor R.L. Stirrat (University of Sussex, UK) and independent consultant Oumoul Khayri Ba Tall, mainly between September and November 2007, with additional input provided by Annelle Bellony (University of Sussex, UK). The OPEV Task Manager was Tony Curran.

Objective

The overall purpose of this evaluation is to inform decisions about the future of the Joint Africa Institute (JAI) and similar capacity-building initiatives.1 The specific objectives as laid down in the Terms of Reference (see Annex 1) are to assess the performance of the JAI in respect of its relevance, effectiveness, and efficiency in delivering policy-related training services and its contribution to building the capacity of African institutions. The evaluation also attempts to assess the organizational effectiveness of JAI and any comparative advantage it may have within the current management and resources context.

Main Findings

• The relevance of the IMF courses to the needs and contexts of the intended beneficiaries is clearly established, given the linkages between IMF policy advice, support programs, and the training delivered.

• The training events supported by the AfDB and the WB reveal an awareness of recent changes in the development environment.

• There is little evidence of a systematic process of needs assessment to identify capacity-building and training priorities.

• There is no evidence of efforts by the JAI’s OC to ensure the development and delivery of a more focused and responsive portfolio of courses and events.

• The JAI management has been severely constrained in promoting synergistic approaches amongst the three member institutions due to the design and funding arrangements of JAI.

• Organizationally, the JAI appears to have been very efficient at ensuring the smooth delivery of training courses and other events

• The JAI has no history of developing and applying strategic planning approaches in the development of its programs or of systematically monitoring and evaluating its overall effectiveness and impact. The JAI also appears to have had no influence on course organizers in terms of ensuring an adequate level of monitoring and evaluation of capacity-building programs.

• The JAI appears to have had little input into the pedagogic approaches used in the courses and events that were delivered through it. Little investment appears to have been made in seeking to use distance training approaches in order to achieve a greater impact with course resources.

• Assessing the cost efficiency of JAI is very problematic, due to the lack of a clear and consistent approach and the paucity of available data on full course costs. JAI’s administrative and financial record-keeping processes are not of the quality required to enable effective management.

• JAI seems to be reasonably cost-efficient. However, costs have been assessed as high by WB and IMF at various times. This is partly explained by the high transportation costs associated with the Tunis location. Cost savings could probably be attained by running more of the JAI programs in alternative African locations.

• The full cost of course delivery is likely to be significantly higher than those reported by JAI, due to the fact that a number of services are provided by the member institutions (especially the AfDB) for which no charges are made.

• Contrary to original expectations, very little synergy was achieved between the institutions in the development and delivery of capacity-building activities.

• Evidence on effectiveness is scarce. End-of-course evaluations are positive but this does not tell us much. Beneficiaries are generally positive about the relevance and utility of the JAI activities to their professional work

• Improvements in individual capacities have been reported as a result of many of the JAI courses/events. However, we do not know if these have been of the scale and nature required to impact institutional performance

• Given the scale, frequency, and type of capacity-building activities conducted by the JAI, it is unlikely that sustained capacity increases in policy development and implementation have been achieved as a result of the training events run by AfDB and WB. It is more likely however, that the IMF courses have achieved the levels of individual capacity improvement required to achieve lasting impact.

• Lacking relevant evidence, this evaluation has been unable to establish the degree to which institutional capacities and performance in development policy-making and implementation have increased as a result of the activities of JAI.

• The nature of IMF policy-strengthening objectives, its trainee selection and follow-up and support processes suggest that significant impact has been achieved through IMF courses, though the extent of this is not known.

• For WB and AfDB courses, it is far less likely that long-lasting impact has been achieved. This is owing to the absence of pre- and post-course support processes to ensure that individual skill-development translates into institutional capacity development.

• There is little evidence that ADI capacity has increased significantly as a result of the JAI’s work.

Main Lessons

• Before training schemes similar to those of the JAI are launched, there must be a clear assessment to identify (i) the nature of developmentally significant gaps in capacity; (ii) how training might help to fill these gaps; and (iii) what sort of training might be most effective.
   
• A results-chain approach needs to be put in place in which the linkages between training activities, trained personnel, increased capacities, and developmental impacts are clearly defined, otherwise the monitoring is limited

• Where there is a collaborative venture involving a number of organizations, there must be clear agreement on objectives. Without this, effective and consistent cooperation among partners is difficult to obtain.

• Monitoring and evaluation mechanisms need to be put in place, in order to assess effectiveness and modify activities so as to increase effectiveness.
   
• Training has to be seen as only one component in an approach to capacity building that recognizes the importance of the wider institutional context in which trainees are placed, and the system of incentives which governs their own and institutional actions.
   
• To be effective, training organizations have to investigate and utilize the potential of new technologies and new methods.
   
• Without efficient and accessible records, it is extremely difficult to make judgments on efficiency. It follows that organizations cannot make cost-efficient choices in the absence of such information.
   
• Without their own financial resources, organizations dependent on external funding are unlikely to be sustainable or able to act in an independent fashion 

Main Recommendations

Recommendation(s) to the Bank:

•  It is strongly recommended that the AfDB formulate a clear and comprehensive policy on capacity building (including training) before it develops further training activities. This should link capacity-building priorities with explicit development objectives, so that training does not become an end in itself. 

• It is recommended that over the next 12 months the AfDB, in collaboration with the two other partner institutions, seriously consider future options for the JAI. These include: • Careful consideration of the plans for the JAI to be integrated with the ADI in 2010 and achieving greater clarity as to what this will involve, in organizational and financial terms; • Assessing the viability of a relaunched JAI with much greater autonomy to develop its own character; and • Clearly analyzing the costs and benefits of closing the JAI at the end of 2009, with a view to maximizing potential gains and managing any negative consequences 

• The policy recommended by the evaluation team is that the JAI should be closed down at the end of 2009 and that the AfDB should develop its own training and capacity-building activities in support of country policies and practices.