Evaluation Team

Authors: Paolo de Renzio, Matt Andrews, Zac Mills.

Sida Evaluation 2010:5
Commissioned by Sida, Secretariat for Evaluation

Published by: Sida, 2010
----

The management group comprises: Joanne Asquith (AfDB); Ole Winckler Andersen (Danida); Nick  Highton and Martin Aldcroft (DFID); and Katarina Kotoglou (Sida).

Objective

This analytical study of quantitative cross-country evidence on Public Financial Management (PFM) in developing countries is part of a broader joint evaluation, initiated by the evaluation departments of Danida, Sida, DFID and the African Development Bank, in consultation with the OECD-DAC Evaluation Network. The evaluation aims to answer two sets of questions:

a) Where and why do PFM reform efforts succeed?

b) Where and how does external support to PFM reform efforts contribute most effectively to their success?

Main Findings

• Economic factors are most important in explaining differences in the quality of PFM systems. Aid-related factors, on the other hand, have more limited explanatory power. As a consequence, PFM systems are more likely to improve responding to changing economic circumstances, rather than to donor efforts.

• More specifically, countries with higher levels of per capita income, with larger populations and with a better recent economic growth record are characterised by better quality PFM systems. On the other hand, state fragility, defined as being in a conflict or post-conflict situation, has a negative effect on the quality of PFM systems.

• Interestingly for the purposes of the evaluation, donor PFM support is also positively and significantly associated with the quality of PFM systems. On average, countries that received more PFMrelated technical assistance have better PFM systems. However, the association is very weak: an additional 40 – 50 million US$ per year would correspond to a half-point increase in the average PEFA score (equivalent to, say, a change from C to C+).

• These results remained consistent through a number of robustness checks and model changes. Interesting additional results come from using more recent data or focusing on low-income countries only. In these cases, the share of total aid provided as general budget support is also positively and significantly associated with better PFM quality. In other words aid modalities, and not just direct support to PFM reforms, contribute to explaining differences in the quality of PFM systems in some of the poorer countries where most donor efforts are concentrated.

• Finally, different aspects of donor support differ in their relationship with more specific PFM processes. A longer period of donor engagement, for example, is associated with better performance in upstream, de jure and concentrated processes. This may be due to donors’ historical tendency to pay more attention to these simpler reform areas, but could also reflect the fact that downstream, de facto and deconcentrated processes take longer to improve.

• The level of donor PFM support is also more strongly associated with scores for de jure and concentrated PFM processes, again highlighting how donor PFM support seems to focus more on rules, procedures and specific actors within government. Results are reversed when it comes to upstream vs. Downstream processes. Here, the association is stronger with downstream processes, possibly highlighting the large amounts of funding devoted to IFMIS projects, a typical downstream PFM reform.

• Data quality remains an issue, especially when it comes to information about donor PFM support. Given the limitations of the information provided by donors, we focused on yearly  disbursements for PFM-related activities. This gives undue weight to large projects such as IFMIS introduction, at the expense of ‘softer’ interventions. We also focused on data post-2002, for which availability is much greater. This means that we cannot capture earlier donor PFM support, when the foundations for PFM reforms were laid in some of the countries included in our sample.

• on one hand and quality of PFM systems on the other is particularly encouraging for the purposes of the evaluation, it clearly cannot be interpreted as causal given the nature of the data. It could merely reflect the fact that donors tend to provide more PFM-related assistance (and more GBS) to countries that have already achieved a certain success in improving the quality  of their PFM systems. Despite various attempts at tackling this issue, we could not prove the direction of causality

• Assessing the impact of donor support on PFM reforms requires tracking the quality of PFM systems over time. Given the lack of sufficient time-series data, the analysis assumes that a higher PEFA score today is a valid proxy for past reform success. Its findings, however, are only partly confirmed by evidence from a smaller dataset looking at changes in PFM systems over the past decade in 19 African countries.