Annex 4. Key Data-Related Findings in Selected IEO Evaluations and IMF Policy Reviews1
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- April 2016
IMF Response to the Financial and Economic Crisis (2014)
While data shortfalls may not have been the main reason the Fund missed the crisis, “the Fund’s analysis of risks and vulnerabilities can, of course, be only as good as the data it is based on” (Robinson, 2014). This notion, put forward by the IEO evaluation on the Response to the Crisis, became particularly relevant during the crisis. The crisis revealed substantial data deficiencies in the realm of risk analysis, showing, for example, that the Fund had too little access to granular banking data. The evaluation also identified the dynamic character of data gaps and how new ones will emerge as financial markets continue to develop and risk analysis becomes more sophisticated. The evaluation concluded that the IMF needed to “take a proactive approach in identifying emerging statistical issues, for instance, through a periodic assessment of the state of global statistics and data gaps most relevant from a global stability perspective for discussion at the Executive Board and the IMFC [International Monetary and Financial Committee].”
IMF Forecasts: Process, Quality, and Country Perspectives (2014)
The IEO evaluation of Forecasts studied the Fund’s macroeconomic predictions. In doing so, it emphasized how the Fund’s forecasting exercises hinge on the quality and timeliness of data. In particular, the evaluation found that data availability was the single most important factor in the choice of forecasting methods, ranking substantially higher than other factors such as historically used methodologies, time constraints, relative accuracy of available alternatives, departmental institutional guidance, or country authorities’ preference. The evaluation report also argued that, as a general rule, the more advanced the economy, the better the quality and availability of data and, therefore, the more room for use of more sophisticated, data-intensive techniques. Among the evaluation’s five recommendations was one critically related to data: data used for forecasts and outturns that already exist internally should be made available to the public (in contrast to the full support for the evaluation’s other four recommendations, this recommendation received only qualified support from Management and the Board).
The Role of the IMF as Trusted Advisor (2013) and IMF Interactions with Member Countries (2010)
The IEO evaluation of Interactions documented how members’ lack of trust affected their data provision to the IMF (a problem raised earlier by the evaluation on Exchange Rates, see below). According to the evaluation surveys, a significant percentage of country authorities (19 percent in large emerging economies, 17 percent in smaller advanced economies, 15 percent in large advanced economies, 14 percent in PRGF-eligible countries, and 7 percent in smaller emerging economies) admitted to withholding data, fearing their possible dissemination to the Executive Board or others.
Along the same lines, the Role of the IMF as Trusted Advisor evaluation analyzed the tension between the roles of the Fund as trusted advisor and ruthless truth-teller or, in other words, between confidentiality and transparency. This trade-off could have a significant impact on the provision of data that authorities consider sensitive. In fact, the evaluation found evidence that authorities in some countries—mainly large emerging markets—were reluctant to have “a candid exchange of views and raising sensitive issues” and noted that “any candor can be used against you.” As the survey to authorities revealed, the ultimate fear was that information shared confidentially may go beyond immediate staff, ranging from other staff and Management to the general public.
Research at the IMF: Relevance and Utilization (2011)
This evaluation found several instances of IMF publications affected by data limitations: (i) Regional Economic Outlooks, where the analysis suffered from the use of data pooled from countries in very diverse circumstances, (ii) Selected Issues Papers, which sometimes did not take into consideration data limitations and used excessively high levels of data aggregation, and (iii) some chapters of the WEO, which based their recommendations on “fragile data.”
IMF Performance in the Run-Up to the Financial and Economic Crisis: IMF Surveillance in 2004–07 (2011)
The IEO evaluation of the IMF Performance in the Run-up to the Crisis identified three data-related problems. First, a significant amount of potentially useful data was ignored or misinterpreted during the period considered. Second, the impact of data issues was asymmetric, suggesting a lack of evenhandedness in the Fund’s interactions with member countries; data limitations did not prevent the IMF from praising the state of some financial systems in advanced economies—including the benefits of risk-diversification—while raising the alarm in some emerging markets. Third, while surveillance teams in advanced countries typically received the information they requested, it was not clear whether they had the capacity to analyze all the information.
The same evaluation found that staff “felt uncomfortable” challenging advanced countries authorities’ views. This was fueled by the assumption that country officials had better access to banking data and knowledge of their financial markets, and by the excellent reputation of central bank economists in these countries. To address these issues, the evaluation suggested enhanced candor and clarity in openly discussing data limitations and methodological qualifications.
IMF Involvement in International Trade Policy Issues (2009)
This evaluation detected how weak data on trade hampered surveillance and generated problems in program design and monitoring. The evaluation also noted a link between data problems and staff resources devoted to data gathering. Case studies revealed that data gathering in the trade area—as in others—is resource intensive, with mission members typically too overburdened to pay sufficient attention. The same was later confirmed by (i) the IEO evaluation report of The Role of the IMF as Trusted Advisor, which found that around 60 percent of mission chiefs agreed that too much of a mission team’s time was devoted to data gathering, reducing the amount of time available for other activities; and by (ii) the evaluation of the Response to the Crisis, which revealed that the effort expended by area department staff to provide, review, and ensure consistency of data across a variety of multilateral surveillance products “seriously impacted their ability to do country work.”
IMF Exchange Rate Policy Advice, 1999–2005 (2007)
This evaluation found that the Fund’s analysis and advice on exchange rate policy was not as effective as it needed to be—due, among other things, to inadequate accuracy, timeliness, and comprehensiveness of data available to staff.2 While data deficiencies were mentioned in several areas, the evaluation identified as particularly problematic for the Fund the reticence of some “big reserves holders” to disclose the composition of their foreign reserves.3 This reticence also prevented these countries from participating in the Currency Composition of Official Foreign Exchange Reserves (COFER) database and the SDDS.
The evaluation also argued that, despite the impact of data deficiencies on the Fund’s operations, evidence suggested that insufficient remedial action had been taken. Staff appeared to have been hesitant to forcefully address identified data problems and prone to certify the adequacy of the data that countries provided. As reasons for this lenience, the evaluation pointed to (i) the convenience of maintaining a smooth relationship with the authorities, and (ii) the absence of sufficient support from Management and the Executive Board for the staff to act more strongly. Moreover, the evaluation raised a possible problem of evenhandedness, since staff seemed to be more reluctant to raise difficult issues with advanced economies, while being more proactive with others. A case in point was the data availability for the 1999 Greek Article IV consultation. The Article IV report itself mentioned that data problems “complicated the assessment of economic conditions.” However, the extent of these deficiencies and their implications were not revealed until much later and, even when uncovered (2004), only a mild reference was included in that year’s Article IV consultation.
Selected IMF Policy Reviews
2014 Triennial Surveillance Review
The 2014 TSR recognized the critical importance of good data for the Fund’s surveillance. It found that IMF mission chiefs regarded lack of data as the most important of the factors inside the Fund that made it harder to do effective surveillance.4
Accordingly, the 2014 TSR attached significant importance to data gaps, making them part of two of its recommendations. Focusing on the Fund’s analysis of risks and spillovers, considered central for Fund surveillance, it acknowledged that enough data were available to perform the core of this type of work, but noted that efforts to further integrate and deepen this analysis would take time, partly because data gaps remained a significant impediment. More specifically, it highlighted two areas:
- External Sector Analysis, where limited data availability is preventing the application of the External Balance Assessment to a larger number of countries, and
- National Balance Sheet Analysis, which could help in detecting risks and understanding how shocks are propagated, but is an area in which “much more progress is needed from the membership to enhance data provision.” For example, IMF staff regrets, more than five years after the collapse of Lehman Brothers, the lack of access to data, even in an aggregated manner, on global systemically important banks and cross-border banking.
The 2014 TSR surveys also identified other areas affected by insufficient data: (i) data constraints are the third most important factor impeding the Fund’s advice on structural issues (after lack of expertise and resource constraints), and (ii) greater availability of comparable cross-country data would be the second most useful initiative, according to staff, in order to improve cross-country analysis in surveillance.
More broadly, the 2014 TSR revealed that the quality of work done by staff is affected by the lack of a “well organized source of information on countries’ experiences,” that goes beyond, but includes, data and statistics (being addressed by the internal work of the Fund on knowledge management). Without such a shared source, knowledge rests with individuals and is often lost. A typical example is the transfer of databases from one country team to the next, which is frequently done improperly, leading to accumulation of errors, inefficiencies, and loss of valuable information. The TSR also mentioned problems with data sharing—including in the use of purchased data—comparability, missing metadata, and lack of resources for data management.
Finally, the 2014 TSR mentions complaints by staff regarding the limited availability of resources for data management. Staff in area departments mentioned during interviews the significant increase in the time absorbed by data and information provision for the production of new multilateral surveillance documents, to be met within the same envelope of resources.
2014 Review of the Financial Sector Assessment Program
This review explained how the effectiveness of stress tests and other analytical work (e.g., on cross-border spillovers) depended fundamentally on the voluntary provision of high-quality data by country authorities. It noted that the reliability of stress tests and the choice of methodology are adversely affected by lack of data, with implications for the comparability of findings across countries. Three data-related constraints were identified as limiting staff’s ability to monitor financial sector risks and to assess financial stability:
- Gaps (both for the IMF and national supervisors) in domestic and cross-border financial data, including data on international interbank markets and the intra-group positions of systemically important financial institutions.
- Uneven access to supervisory data: the provision of bank-by-bank data to FSAP teams remained voluntary under strict confidentiality protocols and was therefore uneven across countries. When authorities do not share the data, especially in advanced economies, the analysis must rely solely on publicly available information or the authorities’ own stress tests, to the detriment of its quality and independence.
- Questions about asset quality: even when authorities share supervisory data, FSAP teams are generally not in a position to assess its accuracy.
After highlighting that data deficiencies were poorly flagged and explained in FSSA reports,5 the review recommended that a more candid judgment of the quality of available data be included in the reports, along with an assessment of the limitations of the analytical results.
2012 Financial Surveillance Strategy
The 2012 Financial Surveillance Strategy also highlighted data gaps as a key challenge to the IMF’s financial surveillance. The strategy called for (i) closer internal attention to the quality of the data provided by members for financial surveillance and (ii) more data on globally systemic financial institutions, to be addressed through participation in a Financial Stability Board group created at the time.
2011 Triennial Surveillance Review
The 2011 TSR identified lack of data as the most important factor impeding surveillance work,6 and included data issues in both its recommendations and operational priorities.
In the area of data, the main focus of the 2011 TSR was on how data issues affect the surveillance of financial sectors. On the one hand, the review recognized that better analysis could be done with the data already available. On the other hand, it highlighted that there are gaps, either because data were not made available to staff or because they did not exist (e.g., on the shadow banking sector). For addressing these gaps, the evaluation put some hope on the Fund’s collaboration agreements with the FSB, but pinpointed legal limitations on sharing individual data as a continuing challenge.
Finally, the 2011 TSR, despite the staff’s concerns regarding data limitations, found that Article IV reports rarely (in five out of fifty cases studied) note financial sector data weaknesses.