Journal Issue
Share
Article

An Evaluation of the World Economic Outlook Forecasts

Author(s):
Allan Timmermann
Published Date:
March 2006
Share
  • ShareShare
Show Summary Details

I. Introduction and Summary

The World Economic Outlook (WEO) is a key source of forecasts of global economic activity and is a key vehicle in the IMF’s multilateral surveillance activities. It is published twice a year in April and September. Given the central role of the WEO forecasts, it is important that they are periodically evaluated to assess their usefulness, and to look for ways to improve the forecasting process. This report is the fourth in a series of such evaluations (following Artis, 1996; Barrionuevo, 1993; and Artis, 1988).

This report analyzes the forecast performance for five key variables—real GDP growth, inflation, the current account balance, and import and export volume growth—from 1990 to 2003, the last year for which actual data were available when the report was initiated. The report incorporates state-of-the-art techniques that shed light on the accuracy of WEO forecasts from new angles, and features three main novel aspects.

  • First, it analyzes forecasts for 178 countries in seven economic regions (Africa, Central and Eastern Europe, the Commonwealth of Independent States (CIS) countries, and Mongolia, developing Asia, the Middle East, the Western Hemisphere, and the advanced economies) since 1990. Earlier evaluations had focused on forecasts for the same variables for only the G-7 countries and regional aggregates.
  • Second, it includes an extensive comparison between the accuracy of WEO forecasts and Consensus forecasts. The latter is a widely used source that compiles the forecasts of economists working in the private sector. Through this comparison, the report assesses WEO forecasts not just against absolute benchmarks, but also against a relative benchmark of other forecasters.
  • Third, it considers the revisions to the forecasts, both over time and within each forecast round. The latter is important because there is a long gestation lag in the preparation of the forecasts in each round, and it is important to know the gains—in terms of accuracy—of frequent forecast updates.

This summary highlights the main findings of the report.

A. How Accurate Are WEO Forecasts?

The first part of the paper examines selected aspects of the WEO forecast performance. In all cases, the analysis considers current-year and next-year forecasts prepared in April and September for a variable. (For example, the April and September 2005 WEOs have projections for 2005 (current year) and 2006 (next year).) Overall, the report finds that WEO forecasts for many variables in many countries meet the basic forecasting quality standards in some, if not all, dimensions. The paper also finds some important issues, which are discussed on a variable-by-variable basis.

  • Real GDP growth. WEO forecasts for real GDP growth display a tendency for systematic overprediction—that is, predicted growth, on average, tends to exceed actual growth (Table S1). From a statistical perspective, these biases are most significant in the next-year forecasts. The results also indicate that systematic overpredictions of real GDP growth are particularly prevalent in forecasts for countries with an IMF program. This tendency for overprediction of growth performance is also persistent over time. The evidence suggests that WEO forecasts for some countries could be improved if more attention were paid to important international linkages. In particular, forecasts of U.S. GDP growth are positively and significantly correlated with current-year forecast errors of output growth in a substantial number of advanced economies. (The forecast of German GDP growth also has predictive power over output growth forecast errors in some regions.) The report also finds that, in some cases, accuracy problems appear related to the standing WEO assumption that the output gap is eliminated after five years. In particular, the paper notes a predominant negative relationship between the output gap and the forecast error in the GDP growth, notably for Germany, France, and Italy.
  • Inflation. The report finds a bias toward underprediction of inflation, with these biases significant in the next-year forecasts for many African, Central and Eastern European, and Western Hemisphere countries. The underprediction bias is generally found to be weaker in the current-year forecasts. With regard to their predictability, there is evidence that the next-year inflation forecast errors are often linked to U.S. GDP forecasts.
  • External current account balances. There appear to be fewer problems in the forecasts for current account balances as percentages of GDP, except for April next-year forecast errors, which, in some cases, are significantly biased or serially correlated. Moreover, general patterns in the direction of biases are not apparent.

Table 1.Overview of Forecast Accuracy(Averages across regions for April current-year forecasts and September next-year forecasts; fraction of economies in region where statistics are significant in parenthesis)
Real GDP GrowthCPI InflationCurrent Account Balance3/
Bias1/Serial

Correlation2/
Bias1/Serial

Correlation2/
Bias1/Serial

Correlation2/
Advanced Economies-0.04/-0.36

(0.07/0.21)
0.21/0.24

(0.03/0.14)
-0.08/-0.13

(0.00/0.07)
0.20/0.33

(10.07/0.17)
0.25/0.22

(0.07/0.11)
0.26/0.35

(0.07/0.29)
Africa-1.17/-1.48

(0.25/0.33)
0.21/0.03

(0.15/0.29)
0.60/1.00

(0.13/0.16)
0.29/0.26

(0.06/0.14)
-0.21/-0.67

(0.02/0.09)
0.27/0.24

(0.13/0.09)
Central and Eastern Europe-1.17/-1.40

(0.07/0.07)
0.37/0.34

(0.13/.20)
2.30/3.27

(0.13/0.16)
0.39/0.39

(0.33/0.33)
20.95/8.50

(0.00/0.00)
0.30/0.33

(0.07/0.13)
Commonwealth of Independent States-1.93/-2.39

(0.15/0.00)
0.31/0.46

(0.00/.15)
126.95/153.65

(0.00/0.20)
0.69/0.74

(0.69/0.77)
-7.49/1.51

(0.00/0.08)
0.24/0.39

(0.08/0.23)
Developing Asia-0.38/-0.53

(0.05/0.04)
0.25/0.31

(0.00/.13)
1.06/0.62

(0.00/0.08)
0.33/0.36

(0.15/0.21)
0.58/1.11

(0.11/0.09)
0.28/0.34

(0.05/0.09)
Middle East-1.66/-1.34

(0.08/0.07)
0.37/0.31

(0.15/.21)
-0.86/-1.15

(0.00/0.08)
0.28/0.27

(0.15/0.00)
-1.36/-1.69

(0.00/0.00)
0.23/0.25

(0.09/0.00)
Western Hemisphere-0.64/-1.16

(0.13/0.24)
0.23/0.21

(0.06/.21)
1.78/1.60

(0.13/0.27)
0.38/0.29

(0.22/0.15)
-0.61/-.95

(0.03/0.16)
0.25/0.27

(0.10/0.16)
Memorandum
G-7 Economies
Canada-0.21/0.58-0.14/0.260.0/-0.20-0.26/0.19-0.01/0.350.43/0.51
France-0.20/-0.80-0.38/-0.230.04/-0.050.32/0.630.14/0.080.38/0.64
Germany-0.13/-1.120.05/-0.260.09/0.120.21/0.37-0.08/-0.350.22/0.46
Italy-0.43/-1.02-0.20/-0.070.08/0.250.08/0.18-0.29/-0.25-0.25/0.38
Japan0.01/-1.03-0.01/0.23-0.07/-0.24-0.19/-0.03-0.02/-0.02-0.06/0.01
United Kingdom-0.02/-0.48-0.60/0.46-0.07/-0.26-0.09/-0.150.33/0.51-0.09/-0.04
United States0.17/0.230.16/0.230.01/-0.30-0.06/0.150.01/-0.12-0.59/-0.16
Source: Author’s calculations.

Mean forecast error.

Coefficient on lagged forecast error in a regression of the forecast error on a constant and its lagged realization.

In percent of GDP.

Source: Author’s calculations.

Mean forecast error.

Coefficient on lagged forecast error in a regression of the forecast error on a constant and its lagged realization.

In percent of GDP.

Besides the basic accuracy of WEO forecasts, the report also examined a number of other issues of interest.

Directional accuracy of forecasts. The results suggest that the WEO forecasts are quite successful in predicting the directional change for current-year real GDP growth and inflation, but somewhat less so for next-year forecasts.

  • Performance of WEO forecasts during the 2001 downturn. WEO forecasts of GDP generally overpredicted growth in 2001 in all regions, which is consistent with the broad patterns among forecasters in earlier downturns. For 2002, the April and September next-year WEO forecasts prepared in 2001 overpredicted growth in six of the seven regions, although revisions in the April 2002 WEO greatly reduced the forecast errors in four regions.

Revisions from board to published forecasts. WEO forecasts are published twice a year in April and September. Prior to publication, a first set of predictions is presented to the IMF Executive Board in February and July. Subsequently, the forecasts are revised before they are published.

These revisions add considerable informational value. For the February/April same-year forecasts, the average reduction in the forecast error is around one-fifth for the advanced economies. The reduction is nearly 30 percent for the July/September same-year forecasts, but only 5 percent for the next-year forecasts.

B. Long-Run Forecasting Performance for G-7 Countries

Taking advantage of the fact that a longer dataset starting in the early seventies exists for the G-7 economies, the report assesses WEO forecasts for these economies in more detail. Overall, the results suggest that the forecast accuracy has deteriorated somewhat since the last evaluation (Artis, 1996). In particular, WEO forecasts systematically and significantly overpredicted economic growth for all the European G-7 economies and Japan during 1991–2003. In contrast, U.S. growth was underpredicted after 1990, although the bias was not found to be statistically significant. In contrast, inflation was strongly and significantly overpredicted for Canada, France, Japan, and the United States during the 1990s and 2000s, although it was underpredicted by a significant margin for Italy.

These findings have at least two possible, not mutually exclusive, explanations. One is that output growth and inflation have been subject to structural breaks, such as a break toward higher productivity growth in the United States. Another possibility is that the underlying assumptions—such as the assumption that the output gap will be eliminated over a 5-year period—has led to biases.

C. Comparison of WEO and Consensus Forecasts

The report compared the WEO projections to Consensus forecasts for GDP growth, inflation, and the current account balance over the period 1990–2003.2 The data covers all the G-7 economies, seven Latin American economies (Argentina, Brazil, Chile, Colombia, Mexico, Peru, and Venezuela) and nine Asian economies (China, Hong Kong SAR, India, Indonesia, Korea, Malaysia, Singapore, Taiwan Province of China, and Thailand).

Overall, the comparison suggests that the forecast performance of the WEO is similar to that of the Consensus forecast—the current-year WEO forecasts of GDP growth in the G-7 economies are generally less biased than the current-year Consensus forecasts, but the bias in the next-year forecasts is stronger in the WEO than in the Consensus across the board. The paper highlights, however, that the timing of the comparison with the Consensus forecast matters. WEO current-year forecasts generally perform quite well against current-year Consensus forecasts reported in March and perform considerably better against the February Consensus forecasts. However, given the relatively long gestation lag in the preparation, they tend to perform considerably worse against the Consensus forecasts reported in April. With the possible exception of next-year inflation forecasts, there is little systematic evidence that the WEO forecasts could be improved by modifying them to account for information embodied in the Consensus forecasts.

D. Recommendations

The report makes the following recommendations to improve the WEO forecasting process:

  • Timeliness of information is key to forecasting performance. There are systematic gains in forecasting accuracy from using the latest available information. These gains were found in both the comparison of the forecasts of the Executive Board version to those of the published version of the WEO and in the comparison between the WEO and the Consensus forecasts. Clearly, the updating process is already adding significant value, especially for the G-7 economies, but more could be done. It is therefore important that IMF country economists update their projections just before publication.
  • Continuous monitoring of forecasting performance. The empirical analysis indicated structural instability in some of the underlying variables, especially real GDP growth and inflation, which is consistent with the broad evidence of instability in macroeconomic variables provided in academic studies. Given the presence of what appears to be systematic biases in forecasting performance for output growth and inflation, particularly after 1990, the possibility of instituting real-time forecasting performance indicators should be explored.
  • Use bias-adjusted forecasts as a guidance. The simplest and most obvious approach to improving forecasts is to shrink the forecast toward its bias-corrected value. While simple to implement if the bias can be precisely estimated, this approach may also be too mechanical and suffer from its own deficiencies, including, for example, the assumption that the bias remains constant through time. Nevertheless, a comparison of unadjusted forecasts with bias-adjusted forecasts can help in enhancing our understanding of the magnitude and direction of any biases that may exist.
  • Forecasts of risk. In view of the inherent uncertainty associated with forecasts, the report strongly recommends that, in the future, the WEO incorporate recent advances in modeling and forecasting risk/uncertainty, including, for example, by presenting the full probability distribution of key variables over time.
  • Review the output gap assumption. The WEO forecasts are based on scenarios that assume that the output gap is removed within a relatively short period. Since some of the countries with the largest output gaps were also found to be countries for which the WEO forecasts systematically overpredicted output growth, this could be a concern. Hence, an analysis that explores the cost and benefits of using this practice is called for. Also, more frequent reviews of estimates of potential output growth may be needed.

The report is organized as follows. Section II describes the principal dataset containing the WEO forecasts and outcomes. Section III introduces the statistical methods used to test the optimality properties of the WEO forecasts. This is followed by two empirical sections that go through the basic characteristics of the forecasts (Section IV) and statistical significance of tests of forecast inefficiencies (Section V). Section VI presents evidence on predictability of WEO forecast errors by means of a range of instruments, such as the WEO prediction of U.S. and German GDP growth, oil prices, the output gap, and the global current account discrepancy. Section VII conducts directional-accuracy tests, while Section VIII studies the information reflected in the process whereby forecasts are revised and updated from their discussion by the Executive Board to the published version of the WEO. Section IX considers the performance of the WEO forecasts during the most recent recession and recovery years; while, conversely, Section X looks at the long-term performance of WEO forecasts for an extended data set on the G-7 countries. Section XI compares the WEO forecasts to the Consensus values, while Section XII analyzes the potential gains from combining these two sets of forecasts. Section XIII looks at recommendations for modifications to the WEO forecasts and forecasting procedures, and Section XIV concludes.

II. Description of Weo Dataset

A. Data Coverage

To assess the forecasting performance, we make use of the fact that four sets of short-term forecasts are available for the same variable since the WEO publishes both April and September current- and next-year forecasts. For example, four forecasts of GDP growth in the year 2000 are reported, namely the April and September 1999 next-year forecasts and the April and September current-year forecasts. Access to different forecast vintages allows us to address issues such as whether (and by how much) the error in the forecast gets reduced as the time towards the target date is shrunk. It also allows us to test another efficiency property embedded in an optimal forecast, namely that forecast revisions should themselves be unpredictable. In some cases we find evidence of significant biases in revisions, suggesting simple ways of improving upon the forecasts.

The WEO data set contains information on 178 countries over the period 1990–2003. These countries are collected into seven groups or regions, namely Africa (50 countries), Central and Eastern Europe (15), CIS and Mongolia (13), Developing Asia (24), Middle East (14), Western Hemisphere (33), and Advanced Economies (29). Data availability and data quality vary significantly across regions and there can be significant differences even within each region. Data quality and the extent to which outliers affect the results also depends on the type of variable being analyzed.

There are five variables for which forecasts are available, namely GDP growth, export volume, import volume, inflation, and current account balance in US dollars or as a percentage of the underlying economies’ GDP. Our analysis focuses on current and next-year forecasts. Both can be considered short-term forecasts. Longer-term forecasts are not further pursued in the analysis because of the rather short data sample which is unlikely to make a statistical analysis of long-term forecasting performance particularly informative.

B. Timing Conventions

Since the target variables are subject to data revisions, a choice has to be made concerning which data vintage to use to measure realized values or outcomes. To this end we follow common practice and use the first-available data in the April WEO issue of year t + 1 to measure the outcome of the predicted variable in period t (labeled yt) while next-year forecasts for period t + 1 are compared to the realized values for year t + 1 (yt+ 1) reported in the September WEO issue of year t + 2.

We will also make use in the analysis of the fact that we have both April and September forecasts of same-year and next-year realizations. This means that we have two sets of current-year forecasts generated in April and September, y^t,t,Apr, y^t,t,Sep, and two sets of next-year forecasts generated during the same months and labeled y^t+1,t,Apr, y^t+1,t,Sep In this notation the first subscript indicates the period being predicted while the second subscript indicates the year when the forecast was generated. The superscript indicates the month of the WEO issue where the WEO forecast was reported. This convention gives rise to four separate forecast errors:

et,tApr=yty^t,tAprApril current year forecast error
et,tSep=yty^t,tSepSeptember current year forecast error
et+1,tApr=yty^t+1,tAprApril current year next forecast error
et+1,tSep=yty^t+1,tSepSeptember next year forecast error

In addition we will also consider current year and next year forecast revisions defined as3

revt,t=y^t,tSepy^t,tAprrevision to the current year forecast
revt+1,t=y^t+1,tSepy^t+1,tAprrevision to the next year forecast.

Table 2 presents basic information on data coverage within each of the regions for the five variables of interest. A maximum of 14 annual data points (1990–2003) is available (13 for next-year forecasts that begin in 1991). The third column reports the number of observations (averaged across countries within each region) that the forecasting analysis makes use of after deleting missing observations and cases where the forecast is exactly identical to the realized value. This leads to a significant trimming of data in some regions. For example, at least eight September current year forecasts are available only for 41 out of 50 African countries and only 11 of the 24 Developing Asian economies had more than eight data points for this variable. Fortunately data on April and September next-year forecasts tend to be more complete although again there are some countries with incomplete data. Measured by data coverage, the data set is most complete for the Advanced Economies and least complete for CIS and Mongolia. Although data coverage does not vary a great deal across variables, the current account data tends to contain somewhat fewer observations.

Table 1.Sample Size, by Variable and Region
Number of CountriesIncludedAverage Number of

Observations per

Country
Real GDP
April current-year forecast errors
Africa504812.52
Central and Eastern Europe151510.80
CIS and Mongolia131310.15
Developing Asia242011.60
Middle East141311.23
Western Hemisphere333211.53
Advanced economies292913.59
September current-year forecast errors
Africa504111.20
Central and Eastern Europe151311.23
CIS and Mongolia131210.17
Developing Asia241111.73
Middle East141010.40
Western Hemisphere332110.86
Advanced economies292912.76
Export Volume
April current-year forecast errors
Africa504712.19
Central Eastern Europe151111.27
CIS and Mongolia13119.73
Developing Asia241611.63
Middle East141411.79
Western Hemisphere333110.77
Advanced economies292813.64
September current-year forecast errors
Africa504913.37
Central and Eastern Europe151410.86
CIS and Mongolia131210.67
Developing Asia242413.08
Middle East141413.07
Western Hemisphere333313.00
Advanced economies292813.57
Import Volume
April current-year forecast errors
Africa504712.21
Central and Eastern Europe151111.27
CIS and Mongolia13119.64
Developing Asia241711.53
Middle East141411.43
Western Hemisphere333110.81
Advanced economies292812.79
September current-year forecast errors
Africa504110.46
Central and Eastern Europe15810.25
CIS and Mongolia1389.75
Developing Asia24812.88
Middle East141011.00
Western Hemisphere331410.50
Advanced economies292812.79
CPI Inflation
April current-year forecast errors
Africa504811.81
Cent. Eastern Europe151511.20
CIS and Mongolia131310.77
Developing Asia242011.35
Middle East141310.46
Western Hemisphere333210.94
Advanced economies292913.24
September current-year forecast errors
Africa505013.48
Cent. Eastern Europe151511.67
CIS and Mongolia131310.77
Developing Asia242412.67
Middle East141413.29
Western Hemisphere333313.36
Advanced economies292913.62
Current Account
April current-year forecast errors
Africa504712.13
Cent. Eastern Europe151510.60
CIS and Mongolia131310.92
Developing Asia241910.68
Middle East141111.73
Western Hemisphere333111.00
Advanced economies292812.50
September current-year forecast errors
Africa504310.51
Cent. Eastern Europe151410.43
CIS and Mongolia131310.62
Developing Asia241111.00
Middle East14810.25
Western Hemisphere331710.29
Advanced economies292812.25
Source: Author’s calculations.
Source: Author’s calculations.
Table 2.Descriptive Statistics for Forecast Errors, by Variable and Region1(Averages across countries in region)
MeanMedianStandard

Deviation
Serial

Correlation
Fraction of

Positive Errors
Real GDP (annual change in percent)
April current-year forecast errors
Africa-1.17-0.813.190.210.34
Central and Eastern Europe-1.17-0.713.490.370.46
CIS and Mongolia-1.93-1.488.280.310.53
Developing Asia-0.38-0.332.220.250.49
Middle East-1.660.206.380.370.53
Western Hemisphere-0.64-0.612.410.230.39
Advanced economies-0.04-0.141.360.210.48
September current-year forecast errors
Africa-0.60-0.522.810.240.40
Central and Eastern Europe-0.110.112.370.330.56
CIS and Mongolia-1.05-0.566.350.270.61
Developing Asia0.160.241.240.270.57
Middle East0.670.223.670.350.58
Western Hemisphere-0.26-0.122.020.240.46
Advanced economies0.09-0.020.810.220.55
April next-year forecast errors
Africa-1.45-1.374.070.280.33
Central and Eastern Europe-1.63-0.923.900.370.39
CIS and Mongolia-2.17-1.938.400.480.51
Developing Asia-0.63-0.672.860.330.45
Middle East-1.060.116.630.320.49
Western Hemisphere-1.33-1.343.080.260.33
Advanced economies-0.55-0.742.060.290.42
September next-year forecast errors
Africa-1.48-1.414.020.230.33
Central and Eastern Europe-1.40-0.973.760.340.41
CIS and Mongolia-2.39-2.789.600.460.52
Developing Asia-0.53-0.682.840.310.45
Middle East-1.340.066.150.310.53
Western Hemisphere-1.16-1.162.960.240.35
Advanced economies-0.36-0.481.970.240.44
Current-year forecast revision
Africa-0.83-0.542.000.230.36
Central and Eastern Europe-0.94-0.452.360.260.51
CIS and Mongolia-1.02-0.855.420.420.53
Developing Asia-0.46-0.511.860.290.45
Middle East-2.33-0.626.170.210.44
Western Hemisphere-0.34-0.271.560.230.38
Advanced economies-0.11-0.070.990.260.49
Next-year forecast revision
Africa-0.08-0.071.550.280.45
Central and Eastern Europe-0.29-0.240.990.310.45
CIS and Mongolia-0.37-0.362.190.240.47
Developing Asia-0.20-0.221.250.280.45
Middle East1.100.274.690.460.47
Western Hemisphere-0.47-0.441.220.310.35
Advanced economies-0.20-0.220.710.210.39
Export Volume (annual change in percent)
April current-year forecast errors
Africa-0.30-1.0520.340.250.43
Central and Eastern Europe-1.32-1.1915.670.290.55
CIS and Mongolia-1.74-0.5533.940.300.49
Developing Asia0.700.688.950.220.51
Middle East-43.59-2.06110.920.260.47
Western Hemisphere-1.70-0.988.510.240.44
Advanced economies-0.39-0.584.830.260.47
September current-year forecast errors
Africa-2.05-0.6315.330.180.44
Central and Eastern Europe0.520.8311.510.370.59
CIS and Mongolia-1.530.0124.540.290.53
Developing Asia0.15-0.025.940.210.52
Middle East0.040.708.570.210.52
Western Hemisphere-0.77-0.506.290.170.51
Advanced economies0.220.213.180.180.55
April next-year forecast errors
Africa-0.16-1.3216.450.240.47
Central and Eastern Europe0.140.7111.800.220.56
CIS and Mongolia-15.76-6.0555.080.340.44
Developing Asia1.842.3111.470.250.56
Middle East-52.750.97121.390.240.51
Western Hemisphere-1.17-0.439.580.240.45
Advanced economies-0.32-0.685.580.240.46
September next-year forecast errors
Africa-0.49-2.0315.210.230.46
Central and Eastern Europe0.160.4811.820.240.55
CIS and Mongolia-6.19-4.0023.320.220.46
Developing Asia0.760.1110.250.270.52
Middle East-27.73-0.22106.900.210.53
Western Hemisphere-1.70-0.908.600.230.43
Advanced economies-0.29-0.325.540.220.49
Current-year forecast revision
Africa1.530.1615.120.190.49
Central and Eastern Europe-1.19-0.8410.270.320.49
CIS and Mongolia-3.41-0.9722.260.290.51
Developing Asia-0.150.014.820.150.52
Middle East-39.210.07104.990.270.46
Western Hemisphere-0.53-0.224.330.170.46
Advanced economies-0.48-0.543.390.270.41
Next-year forecast revision
Africa0.690.378.150.180.51
Central and Eastern Europe0.410.092.960.270.54
CIS and Mongolia-7.72-1.0448.640.230.48
Developing Asia0.230.062.560.190.51
Middle East-20.320.6077.690.220.51
Western Hemisphere0.470.374.470.200.55
Advanced economies-0.07-0.171.780.210.48
Import Volume (annual change in percent)
April current-year forecast errors
Africa-0.610.2515.290.220.46
Central and Eastern Europe-2.47-1.0720.290.380.49
CIS and Mongolia-1.15-1.6036.530.310.50
Developing Asia-0.150.108.680.310.53
Middle East-42.400.63138.340.330.49
Western Hemisphere0.990.9610.770.300.52
Advanced economies-0.29-0.465.390.230.48
September current-year forecast errors
Africa-2.47-3.3014.610.270.43
Central and Eastern Europe-0.64-0.6817.720.400.58
CIS and Mongolia0.632.8936.330.310.50
Developing Asia-0.51-0.537.340.310.47
Middle East1.732.0819.850.320.54
Western Hemisphere0.631.339.910.370.53
Advanced economies0.280.333.920.210.53
April next-year forecast errors
Africa0.09-0.2515.220.220.47
Central and Eastern Europe-0.311.5117.300.270.54
CIS and Mongolia-4.00-3.0723.620.300.48
Developing Asia0.741.0711.300.180.56
Middle East-56.451.62136.470.160.52
Western Hemisphere0.960.1212.440.300.52
Advanced economies-0.13-0.726.450.220.50
September next-year forecast errors
Africa0.05-0.1014.150.210.47
Central and Eastern Europe2.281.6413.210.270.58
CIS and Mongolia-1.52-1.1219.350.350.48
Developing Asia0.340.5610.990.150.55
Middle East-37.350.39122.320.240.50
Western Hemisphere1.021.2511.880.280.51
Advanced economies-0.02-0.286.270.210.51
Current-year forecast revision
Africa0.871.1014.690.230.50
Central and Eastern Europe-2.64-2.9911.490.370.41
CIS and Mongolia0.99-0.8517.070.230.53
Developing Asia-0.45-0.356.460.240.47
Middle East-73.03-0.58209.850.220.55
Western Hemisphere0.900.298.960.260.55
Advanced economies-0.47-0.623.740.210.47
Next-year forecast revision
Africa0.870.399.090.260.51
Central and Eastern Europe-1.880.0610.320.330.60
CIS and Mongolia-2.020.5320.780.230.65
Developing Asia0.510.114.330.300.56
Middle East-30.49-0.65115.140.320.57
Western Hemisphere0.690.057.180.270.47
Advanced economies-0.18-0.162.110.220.44
Inflation (in percent per year)
April current-year forecast errors
Africa57.480.60162.180.290.57
Central and Eastern Europe7.362.3024.370.390.53
CIS and Mongolia340.63126.95978.650.690.49
Developing Asia1.721.068.200.330.53
Middle East-2.03-0.868.490.280.34
Western Hemisphere18.691.7850.400.380.56
Advanced economies-0.08-0.030.940.200.44
September current-year forecast errors
Africa39.700.12133.390.230.54
Central and Eastern Europe2.030.178.790.270.47
CIS and Mongolia163.1664.00554.430.600.42
Developing Asia1.000.095.970.230.47
Middle East-0.59-0.418.610.190.43
Western Hemisphere7.060.5823.870.270.50
Advanced economies-0.09-0.050.490.230.42
April next-year forecast errors
Africa81.722.50177.600.260.66
Central and Eastern Europe16.054.0334.060.410.60
CIS and Mongolia229.71177.67592.490.830.70
Developing Asia1.451.289.160.420.55
Middle East-0.77-0.9611.320.380.38
Western Hemisphere10.852.6662.080.380.59
Advanced economies-0.12-0.131.430.360.43
September next-year forecast errors
Africa74.941.80164.020.260.62
Central and Eastern Europe16.043.2732.430.390.58
CIS and Mongolia190.99153.65590.620.740.65
Developing Asia1.630.629.150.360.52
Middle East-1.77-1.1510.960.270.32
Western Hemisphere7.811.6054.870.290.58
Advanced economies-0.19-0.131.200.330.40
Current-year forecast revision
Africa17.030.2738.110.220.56
Central and Eastern Europe5.001.7819.150.200.58
CIS and Mongolia182.11120.45551.080.610.56
Developing Asia0.310.153.210.290.52
Middle East0.19-0.013.180.340.51
Western Hemisphere9.670.2929.170.180.51
Advanced economies0.030.030.760.190.51
Next-year forecast revision
Africa3.920.3011.010.210.57
Central and Eastern Europe1.600.697.300.150.63
CIS and Mongolia36.8311.26112.550.420.69
Developing Asia0.220.222.780.190.56
Middle East0.740.403.840.340.57
Western Hemisphere3.200.0712.270.180.57
Advanced economies0.050.010.700.260.53
Current Account (in percent of GDP)
April current-year forecast errors
Africa-0.210.355.700.270.52
Central and Eastern Europe20.950.5069.310.300.48
CIS and Mongolia-7.490.0395.360.240.61
Developing Asia0.580.425.780.280.58
Middle East-1.36-0.2412.980.230.61
Western Hemisphere-0.61-0.184.170.250.46
Advanced economies0.250.101.780.260.52
September current-year forecast errors
Africa0.170.255.250.250.54
Central and Eastern Europe4.300.5814.560.270.52
CIS and Mongolia-21.04-1.4782.000.390.54
Developing Asia1.280.604.690.280.63
Middle East-0.53-0.296.160.270.56
Western Hemisphere0.17-0.122.670.290.51
Advanced economies0.030.051.330.260.51
April next-year forecast errors
Africa-1.04-0.216.730.280.46
Central and Eastern Europe34.91-0.19110.890.390.41
CIS and Mongolia40.181.50128.560.340.59
Developing Asia0.780.677.640.400.56
Middle East0.301.8218.520.310.65
Western Hemisphere-1.52-0.835.360.300.39
Advanced economies0.390.092.420.380.52
September next-year forecast errors
Africa-0.670.206.880.240.50
Central and Eastern Europe8.50-0.7732.260.330.46
CIS and Mongolia1.510.6211.530.390.57
Developing Asia1.110.287.160.340.58
Middle East-1.691.0417.850.250.59
Western Hemisphere-0.93-0.504.990.270.42
Advanced economies0.220.092.250.350.51
Current-year forecast revision
Africa-0.29-0.254.690.220.50
Central and Eastern Europe18.030.1862.400.260.44
CIS and Mongolia16.210.9962.230.350.64
Developing Asia0.100.192.750.240.56
Middle East1.581.125.370.220.66
Western Hemisphere0.130.101.730.280.48
Advanced economies0.220.131.180.260.56
Next-year forecast revision
Africa-0.31-0.404.600.270.48
Central and Eastern Europe29.28-0.0499.550.220.39
CIS and Mongolia42.120.26135.870.330.64
Developing Asia-0.010.082.320.260.58
Middle East2.172.245.320.240.69
Western Hemisphere0.150.041.670.220.52
Advanced economies0.170.151.230.220.52

Except for the median which is the median of the mean forecast error across countries.

Except for the median which is the median of the mean forecast error across countries.

III. Properties of Optimal Forecasts

To evaluate the quality of the WEO forecasts it is necessary to establish a set of testable properties that an optimal forecast should have. In this section we discuss the nature of such properties. In all cases the properties are established under the assumption that the objective function is of the mean squared error (MSE) type so the forecasts minimize a symmetric, quadratic loss function. Different properties hold for other loss functions. For a theoretical discussion and derivation of these properties, see Patton and Timmermann (2004).

A. Unbiasedness and Lack of Serial Correlation

Most fundamentally, an optimal forecast should be unbiased and serially uncorrelated. Define the generic forecast errors for period t or t + 1, computed at time τ, as

To test the basic unbiasedness and uncorrelatedness properties, one can perform simple regressions

For an efficient forecast we must have α = 0 (unbiasedness) in (1) and α = 0, β = 0 in (2), implying unbiasedness and absence of serial correlation. The first regression gives rise to a simple student-t test of α = 0, while the second leads to an F-test. Adding the forecast, ŷt+ 1, τ, to both sides of equation (2), this regression is easily seen to be equivalent to the conventional Mincer-Zarnowitz (1969) levels regression

In this regression unbiasedness of the forecast translates into a requirement that α = 0, β = 1.

B. Efficiency Properties More Generally

Unbiasedness and absence of serial correlation in the forecast errors can be thought of as weak efficiency requirements. A much more general and stricter orthogonality condition holds for optimal forecasts under MSE loss. Since an optimal forecast should be the conditional expectation of the predicted variable of interest, if the forecaster uses all available information efficiently, then no variable in the current information set should be able to predict future forecast errors. To test this, let zt be any such variable in the forecaster’s information set at time t, Ωt. An implication of informational efficiency is that α = β = 0 in the regression

where εt+1 is a serially uncorrelated, zero-mean error term. The relationship between unbiasedness and absence of serial correlation on the one hand (equation (2)) and informational efficiency according to (4) more generally is similar to the relationship between the weak and semi-strong versions of the market efficiency hypothesis. According to the weakly efficient hypothesis, past values of the variable itself should not help predict future values. The semi-strong version tightens this restriction by requiring that no publicly available information should help forecast future values.

C. Forecast Revisions: Efficiency Tests Without Measurement Problems

Forecast revisions are of fundamental interest in a forecast evaluation exercise for one simple reason: If a sequence of forecasts is optimal, then the forecast revisions should themselves be unpredictable (technically a martingale difference sequence).

Indeed, if this were not the case and, say, forecast revisions between April and September were themselves predictable, then the original (April) forecast would not be optimal. Suppose, for example, that it is known that on average the September forecast of next-year output growth tends to be ¼ of 1 percent higher than the April forecast. Then the April forecast should be revised upwards by this amount to reflect the better information available in September of each year.

Another advantage of studying revisions is that predictable patterns in revisions, if detected, automatically tells the forecaster how to improve the original forecast, namely by amending it by the fitted value of the forecast revision. Hence if the April forecast of the revision in the forecast between April and September is

then the original April forecast, y^t,tApr can be replaced by an improved forecast, y˜t,tApr, as follows:

More generally, if ΩtSep is the forecaster’s information set in September, ΩtApr is the information set in April—which is a sub-set of the September information set, ΩtAprΩtSep — and if forecasts are formed optimally as conditional expectations, i.e.

then by the law of iterated expectations E[y^t+1,tSep|ΩtApr]=y^t+1,tApr and so the revision, defined as revt+1,t=y^t+1,tSepy^t+1,tApr must be zero-mean:

A similar result holds for the current-year revisions, revt,t=y^t,tSepy^t,tApr:

Notice, however, that in general E[revt+1,t|ΩtSep]0 and E[revt,t|ΩtSep]0 provided that any new information of use to the forecaster arrives between April and September of year t.4 It is worth pointing out that we ignore the effect of estimation errors, which can induce serial correlation in the forecast errors even if the forecaster knows the true model. This is akin to learning effects—see Timmermann (1993) for a discussion of this point in the context of predictability of financial returns.

An important implication follows from these simple results: Forecast optimality can be tested without having data on the target variable, y. This is important since, given the availability of different vintages of the target variable, it is not clear whether the forecasts should be compared to first-issue, second (revised) or the “final” data revision. This matters considerably in practice as witnessed by the recent literature on “real-time” macroeconomic data, (see Croushore, 2005). By analyzing data revisions we can effectively construct a test that is not sensitive to how well the underlying data is being measured.

D. Non-Increasing Variance of Forecast Errors as Forecast Horizon Is Decreased

A final property that an optimal forecast should have is that the variance of the forecast error should decline as more information becomes available. This means that the April current year (next year) forecast errors should have a greater variance than the September current-year (next year) forecast errors:

Intuitively this simply reflects that more information about the outcome in the current or next year is known in September than in April of the same year. This can be formally tested through a variance ratio test or (more appropriately given the small sample size here) by considering patterns in the variance of forecast errors associated with different forecast horizons.

IV. Empirical Results

With the dataset and benchmark properties of an optimal forecast in place, we proceed to analyze the empirical evidence. Table 3 reports summary statistics for the forecast errors and forecast revisions grouped by variable and region. We show the mean, median, and standard deviation of the forecast error, the average absolute value of the coefficient of first-order serial correlation in the forecast errors and the percentage of positive values of the forecast error. In all cases these statistics are computed based on the cross-section of countries within a particular region. For example, the median value is the median of the mean values across countries in a given region, while the standard deviation is computed across the mean values for countries in the region. In the sequel we discuss the main empirical findings.

Table 3.Tests for Biasedness and Serial Correlation of Forecast Errors(Share of countries in region with significant test statistics)
Forecast Error Bias (α^)1/Serial Correlation (β^)3/
T-Value for α^Fraction of

Bootstrap

P-Value

<0.052/
|tβ^|>2Fraction of

Bootstrap

P-Value

<0.054/
Percent of

Significant

Sign Tests 5/

(P-Value <.05)
tα^<2tα^>2
Real GDP
April current-year forecast errors
Africa0.400.000.250.060.150.17
Central and eastern Europe0.130.000.070.130.130.20
CIS and Mongolia0.080.080.150.080.000.15
Developing Asia0.100.050.050.000.000.15
Middle East0.000.080.080.310.150.08
Western Hemisphere0.190.000.130.030.060.03
Advanced economies0.030.030.070.000.030.10
September current-year forecast errors
Africa0.200.000.150.120.150.15
Central and eastern Europe0.000.080.000.150.000.08
CIS and Mongolia0.000.170.080.170.080.17
Developing Asia0.000.090.000.000.000.18
Middle East0.000.200.200.300.200.10
Western Hemisphere0.100.050.050.050.000.05
Advanced economies0.000.100.070.070.100.17
April next-year forecast errors
Africa0.380.040.330.150.330.35
Central and eastern Europe0.270.000.070.270.270.07
CIS and Mongolia0.150.000.000.460.080.08
Developing Asia0.170.040.090.300.130.09
Middle East0.000.070.000.140.140.07
Western Hemisphere0.360.000.270.090.150.18
Advanced economies0.280.030.240.140.210.14
September next-year forecast errors
Africa0.400.020.330.130.290.33
Central and eastern Europe0.200.000.070.200.200.13
CIS and Mongolia0.080.000.000.380.150.08
Developing Asia0.040.040.040.170.130.09
Middle East0.000.070.070.140.210.07
Western Hemisphere0.330.000.240.060.210.09
Advanced economies0.240.070.210.000.140.10
Current-year forecast revision
Africa0.290.000.170.100.100.12
Central and eastern Europe0.180.000.000.000.000.09
CIS and Mongolia0.000.000.000.200.000.00
Developing Asia0.080.000.080.080.080.08
Middle East0.000.000.000.100.000.00
Western Hemisphere0.200.000.130.000.000.00
Advanced economies0.040.040.040.070.040.11
Next-year forecast revision
Africa0.070.000.020.100.050.07
Central and eastern Europe0.220.000.000.220.110.00
CIS and Mongolia0.000.000.000.000.000.00
Developing Asia0.080.000.080.000.000.00
Middle East0.000.000.000.380.000.00
Western Hemisphere0.200.000.130.070.130.07
Advanced economies0.190.000.070.070.070.11
Export Volume
April current-year forecast errors
Africa0.090.000.090.040.060.06
Central and eastern Europe0.000.090.000.180.000.09
CIS and Mongolia0.000.000.000.180.090.00
Developing Asia0.000.060.060.000.000.06
Middle East0.070.000.000.140.070.07
Western Hemisphere0.060.000.030.030.030.06
Advanced economies0.000.040.040.140.040.04
September current-year forecast errors
Africa0.100.000.040.040.020.04
Central and eastern Europe0.000.070.070.210.070.14
CIS and Mongolia0.000.000.000.170.000.08
Developing Asia0.000.040.000.040.040.08
Middle East0.000.070.000.070.000.07
Western Hemisphere0.000.000.000.030.030.09
Advanced economies0.000.040.000.070.040.11
April next-year forecast errors
Africa0.070.020.020.020.000.04
Central and eastern Europe0.000.000.000.000.000.23
CIS and Mongolia0.080.000.000.150.000.08
Developing Asia0.000.090.000.050.000.18
Middle East0.000.070.070.000.000.14
Western Hemisphere0.030.000.000.030.000.00
Advanced economies0.040.070.040.040.040.07
September next-year forecast errors
Africa0.040.020.040.000.020.06
Central and eastern Europe0.000.000.000.080.000.17
CIS and Mongolia0.080.000.000.000.000.08
Developing Asia0.000.040.040.130.000.08
Middle East0.000.140.140.000.000.07
Western Hemisphere0.060.000.060.030.030.00
Advanced economies0.040.040.000.070.040.07
Current-year forecast revision
Africa0.040.000.000.020.020.10
Central and eastern Europe0.000.000.000.140.070.14
CIS and Mongolia0.000.000.000.230.150.15
Developing Asia0.040.000.000.000.000.08
Middle East0.000.000.000.070.000.00
Western Hemisphere0.090.000.000.060.030.06
Advanced economies0.070.000.040.110.000.07
Next-year forecast revision
Africa0.040.020.020.040.060.04
Central and eastern Europe0.000.000.000.070.000.00
CIS and Mongolia0.000.080.000.080.000.08
Developing Asia0.000.040.000.000.000.00
Middle East0.000.070.000.070.000.14
Western Hemisphere0.000.030.000.060.030.03
Advanced economies0.070.040.070.070.070.04
Import Volume
April current-year forecast errors
Africa0.040.020.020.040.060.06
Central and eastern Europe0.000.000.000.360.090.00
CIS and Mongolia0.000.000.000.180.000.09
Developing Asia0.000.060.000.120.120.12
Middle East0.000.000.000.290.140.00
Western Hemisphere0.030.000.000.030.030.00
Advanced economies0.000.000.000.040.000.04
September current-year forecast errors
Africa0.070.000.070.070.070.07
Central and eastern Europe0.000.000.000.130.130.13
CIS and Mongolia0.000.000.000.130.000.00
Developing Asia0.000.000.000.000.000.00
Middle East0.000.000.000.100.100.00
Western Hemisphere0.000.000.000.140.140.00
Advanced economies0.000.000.000.040.000.07
April next-year forecast errors
Africa0.040.040.020.020.020.04
Central and eastern Europe0.000.080.000.000.000.00
CIS and Mongolia0.080.000.000.000.080.08
Developing Asia0.000.000.000.090.000.09
Middle East0.070.000.000.000.000.14
Western Hemisphere0.000.030.000.090.030.06
Advanced economies0.000.000.000.040.040.07
September next-year forecast errors
Africa0.020.040.040.090.020.09
Central and eastern Europe0.000.100.100.100.000.10
CIS and Mongolia0.090.000.000.090.000.00
Developing Asia0.000.000.000.050.000.09
Middle East0.070.000.000.000.000.07
Western Hemisphere0.000.030.030.060.060.03
Advanced economies0.000.000.000.040.000.07
Current-year forecast revision
Africa0.030.030.030.050.050.07
Central and eastern Europe0.000.000.000.110.000.00
CIS and Mongolia0.000.000.000.000.000.14
Developing Asia0.000.000.000.180.090.00
Middle East0.000.000.000.110.000.00
Western Hemisphere0.000.000.000.000.000.08
Advanced economies0.140.070.180.000.110.14
Next-year forecast revision
Africa0.030.050.000.030.000.10
Central and eastern Europe0.000.000.000.220.000.00
CIS and Mongolia0.000.000.000.140.000.14
Developing Asia0.000.000.000.000.000.00
Middle East0.000.000.000.110.000.00
Western Hemisphere0.000.000.000.090.000.00
Advanced economies0.180.000.140.070.070.07
Inflation
April current-year forecast errors
Africa0.000.190.130.060.130.21
Central and eastern Europe0.000.070.000.330.070.13
CIS and Mongolia0.000.080.000.690.310.00
Developing Asia0.000.050.000.150.000.15
Middle East0.080.000.000.150.000.15
Western Hemisphere0.030.190.130.220.090.16
Advanced economies0.070.030.000.070.070.07
September current-year forecast errors
Africa0.040.080.000.060.040.16
Central and eastern Europe0.070.000.000.200.000.00
CIS and Mongolia0.000.080.000.620.150.00
Developing Asia0.080.000.080.040.080.13
Middle East0.000.000.000.000.000.07
Western Hemisphere0.030.060.030.120.060.06
Advanced economies0.170.000.030.070.030.00
April next-year forecast errors
Africa0.000.250.190.190.170.31
Central and eastern Europe0.000.200.130.330.200.33
CIS and Mongolia0.000.310.000.920.310.31
Developing Asia0.000.170.130.350.260.26
Middle East0.290.140.140.290.210.36
Western Hemisphere0.150.270.240.330.390.36
Advanced economies0.140.030.140.100.170.10
September next-year forecast errors
Africa0.020.220.160.140.140.34
Central and eastern Europe0.000.270.200.330.200.40
CIS and Mongolia0.000.230.080.770.150.23
Developing Asia0.130.130.080.210.170.25
Middle East0.210.140.140.000.140.21
Western Hemisphere0.120.240.270.150.360.42
Advanced economies0.070.000.070.170.140.07
Current-year forecast revision
Africa0.000.140.020.080.020.18
Central and eastern Europe0.000.070.000.200.000.07
CIS and Mongolia0.000.080.000.620.310.15
Developing Asia0.000.080.040.080.040.17
Middle East0.000.000.000.290.000.21
Western Hemisphere0.000.120.000.060.000.12
Advanced economies0.030.030.070.070.030.07
Next-year forecast revision
Africa0.020.080.040.060.060.18
Central and eastern Europe0.000.130.070.000.000.20
CIS and Mongolia0.000.000.000.380.080.38
Developing Asia0.000.170.040.090.090.13
Middle East0.000.000.000.210.070.07
Western Hemisphere0.000.060.030.060.030.24
Advanced economies0.030.100.030.140.100.10
Current Account (in percent of GDP)
April current-year forecast errors
Africa0.020.020.020.130.020.04
Central and eastern Europe0.000.000.000.070.070.00
CIS and Mongolia0.000.080.000.080.000.08
Developing Asia0.000.160.110.050.050.16
Middle East0.000.090.000.090.090.18
Western Hemisphere0.030.000.030.100.030.06
Advanced economies0.000.070.070.070.040.07
September current-year forecast errors
Africa0.000.090.020.050.050.09
Central and eastern Europe0.000.000.000.070.000.07
CIS and Mongolia0.000.080.080.230.080.23
Developing Asia0.000.180.180.090.090.36
Middle East0.000.000.000.000.000.13
Western Hemisphere0.060.060.060.000.060.00
Advanced economies0.000.040.000.110.040.11
April next-year forecast errors
Africa0.130.020.130.040.090.11
Central and eastern Europe0.070.000.070.200.130.13
CIS and Mongolia0.000.230.230.150.150.46
Developing Asia0.000.130.090.260.260.17
Middle East0.000.140.140.140.140.36
Western Hemisphere0.240.030.270.090.240.12
Advanced economies0.040.180.140.320.180.14
September next-year forecast errors
Africa0.090.040.090.090.020.13
Central and eastern Europe0.070.000.000.130.130.07
CIS and Mongolia0.080.150.080.230.000.15
Developing Asia0.000.130.090.090.040.17
Middle East0.000.070.000.000.070.14
Western Hemisphere0.160.030.160.160.160.03
Advanced economies0.040.140.110.290.070.14
Current-year forecast revision
Africa0.020.040.070.040.000.09
Central and eastern Europe0.130.000.000.130.000.07
CIS and Mongolia0.000.080.000.170.080.42
Developing Asia0.000.080.000.080.080.15
Middle East0.000.180.000.000.000.09
Western Hemisphere0.000.000.000.000.000.06
Advanced economies0.000.040.040.070.040.04
Next-year forecast revision
Africa0.070.020.020.110.070.04
Central and eastern Europe0.070.000.000.070.000.07
CIS and Mongolia0.000.080.000.250.080.25
Developing Asia0.000.000.000.000.000.08
Middle East0.000.180.090.000.000.36
Western Hemisphere0.000.000.000.060.000.06
Advanced economies0.040.040.000.000.000.00
Source: Author’s calculations.

The bias coefficient α is defined in equation (1) in the main text.

Fraction of bootstrapped p-values for the null hypothesis α^=0 that are smaller than 0.05 in a two-sided test.

The serial correlation coefficient β is defined in equation (2).

Fraction of bootstrapped p-values for the F-test of the joint null hypothesis of α = 0 and β = 0 that are smaller than 0.05.

Fraction of significant test values (p-value of less than or equal to 0.05) for a test of the null hypothesis that the fraction of positive forecast errors equals 0.5.

Source: Author’s calculations.

The bias coefficient α is defined in equation (1) in the main text.

Fraction of bootstrapped p-values for the null hypothesis α^=0 that are smaller than 0.05 in a two-sided test.

The serial correlation coefficient β is defined in equation (2).

Fraction of bootstrapped p-values for the F-test of the joint null hypothesis of α = 0 and β = 0 that are smaller than 0.05.

Fraction of significant test values (p-value of less than or equal to 0.05) for a test of the null hypothesis that the fraction of positive forecast errors equals 0.5.

A. GDP Growth

Current-year forecasts

For the real GDP growth rate variable the mean of the current-year forecast error (i.e., the bias averaged across time and across countries) is very close to zero for the Advanced Economies. Biases in April current-year forecasts are much larger—exceeding more than one percent—and negative for Africa, Central and Eastern Europe, CIS and Mongolia, and the Middle East. As expected, this bias is reduced significantly in the September current-year forecasts. Although the April biases appear to be rather large, it should also be noted that they reflect some very large outliers whose values are predominantly negative and thus represent overpredictions. Indeed, the standard deviations of the April current-year forecast errors tend to be largest for those regions where the greatest biases were found—exceeding 8 percent for CIS and Mongolia and 6 percent for the Middle East.

Due to the presence of such outliers in the data we also consider more robust statistics such as the median forecast error and the proportion of positive forecast errors (underpredictions).

Provided that the underlying shocks are not drawn from asymmetric distributions, one would expect the median to be close to zero and the proportion of positive forecast errors to be close to 50 percent on average if the underlying forecasting model is not misspecified. Again the data reveals systematic problems for some of the regions: between 34 and 40 percent of the same-year forecasts for the African region are overpredictions of subsequent GDP growth (negative mean forecast errors). Consistent with this, the median forecast error remains large and negative (-0.81 for this region) as it does for Central and Eastern Europe and CIS and Mongolia.

Forecasts in all regions pass the test that the variance of the September forecast errors should be smaller than the variance of the April forecast errors of the same variable. Furthermore, in many regions the reduction in uncertainty between the April and September forecast appears to be quite large. For example, the average standard deviation of the current-year forecast error in the Advanced Economies is reduced from 1.36 percent in April to 0.81 percent in September, representing a 40 percent reduction.

Next-year forecasts

Biases in the next-year forecast errors generally exceed those observed in the current-year forecasts. Interestingly, in every single region the mean April or September biases are negative and this also holds for the median bias in all regions with exception of the Middle East. This suggests that the WEO in general overpredicts next-year GDP growth. Furthermore, whereas the average bias in the current-year predictions for the Advanced Economies is very small, it is quite sizeable in the next year forecast where it takes values of -0.36 and -0.55 percent, depending on the reporting date of the forecast. Estimates of the standard deviations of the forecast errors associated with the April and September next year forecasts are much more similar than their current year counterparts. This suggests that far less is learned between April and September about next-year growth than is learned between these months about growth in the current year.5

The proportion of positive next-year forecast errors is again very low for Africa (0.33) and the Western Hemisphere (0.35). The predominance of regions with proportions of positive signs below 0.5 is consistent with the tendency of the WEO forecasts to over-predict next-year GDP growth.

Serial correlation in the forecast errors appears to also be a problem in some regions. The fourth column of Table 3, which reports the average of the absolute value of the first-order autocorrelation in the forecast error is quite high in central and eastern Europe and CIS and Mongolia in particular.

Turning to the forecast revisions between the April and September WEO publications, which should have a mean of zero, there is systematic evidence of negative biases. This is consistent with the April and September forecasts both overpredicting GDP growth on average, but the April forecast being more optimistic than the September value (so the mean change is negative). Hence on average the September forecast is being revised downwards when compared to the April value. This finding is corroborated in the median values as well as in the proportion of positive forecast revisions (which consistently lies below one-half) and is information that could easily be used to improve upon the WEO growth forecasts.

Another feature that is worth noting in the forecast revisions is that the standard deviation of the revision is generally quite a bit larger for the current-year values than for the next-year values. Again, this reinforces the earlier observation that information arriving between April and September more strongly affects current-year than next-year forecasts.

B. Inflation

Very high inflation rates characterized a number of countries during the sample period so it is unsurprising that outliers tend to be very large for this variable and certainly larger than for real GDP growth. As a consequence we focus our analysis on the relatively robust measures of forecasting performance such as the proportion of positive forecast errors. For the current-year forecasts this does not deviate too strongly from 50 percent in any of the regions, except for the Middle East, where only between 34 and 43 percent of the April and September current-year forecast errors are positive, and to a lesser extent for the Advanced Economies where 43 percent of the signs are positive.

A rather different picture emerges for the next-year forecast errors. Between 60 and 70 percent of the April forecast errors are positive for Africa, central and eastern Europe and CIS and Mongolia. These proportions are closer to 60 percent for the September forecasts, but remain somewhat higher than 50 percent, indicating a tendency toward underprediction of inflation in these countries. Furthermore, all forecast revisions have positive means and more than 50 percent of the forecast revisions are positive. A particularly high percentage is observed among the next-year revisions for CIS and Mongolia and central and eastern Europe which generally see the average forecast revised upward. Hence there is a tendency for both the WEO’s current-year and next-year inflation forecasts to be raised between April and September. Since the September forecasts are generally more accurate than their April counterparts, this suggests that the April WEO inflation forecasts can be improved by increasing their value.

We also consider whether the standard deviation of the April forecast errors is greater than that of the September forecast errors. Although outliers make it difficult to interpret some of the values, this appears generally to be the case.

C. Export and Import Volumes

Data on export and import volume is also affected strongly by outliers. Outliers in export volume with values in triple digits affect Africa, CIS and Mongolia and the Middle Eastern region, whereas central and eastern Europe, developing Asia, Western Hemisphere and the Advanced Economies are not so strongly influenced by these. For the regions not affected by outliers, the bias appears rather modest—ranging between 0.22 and -0.39 for the Advanced Economies, for example. Furthermore, the September current-year and next-year forecast errors generally have a smaller standard deviation than the corresponding April values, suggesting that information arriving between April and September is used to improve upon the forecasts. For the vast majority of regions and forecast horizons the proportion of positive forecast errors (or revisions) is close to 50 percent as one would expect for a weakly efficient forecast under a symmetric error distribution.

For the import volume data only the Advanced Economies, Western Hemisphere and developing Asia are not affected by triple-digit outliers. Average biases are small in economic terms and at or below one percent in absolute value for these regions. In all regions we continue to see a proportion of positive forecast errors that is quite close to 50 percent.

D. Current Account Balance

Current account balance figures measured in U.S. dollar terms or as a percentage of GDP suggest a slight tendency for the WEO current-year forecasts to under predict, particularly in central and eastern Europe and developing Asia. Forecast revisions mostly have positive means so the September forecast is generally above the April value, again indicating a simple method for improving upon the quality of the April forecasts.

V. Analysis of Statistical Significance

Whether the biases documented in the previous table should be of concern depends on how systematic they are. This issue can best be addressed by undertaking a more in-depth statistical analysis. Such an analysis is of course tempered by the short data sample which potentially invalidates inference relying on asymptotic distributions but also lowers the power of a statistical analysis to detect misspecification in the forecasting models even when this is present. Again countries with fewer than eight observations will be excluded from the statistical analysis. Considerable caution should be exercised when interpreting the statistical inference results since the sample size used here is very small, and finite-sample distortions of standard test-statistics that correct for heteroskedasticity and autocorrelation in the regression residuals are well known (see, for example, Den Haan and Levin, 1997; Kiefer, Vogelsang, and Bunzel, 2000; and Kiefer and Vogelsang, 2005).

To deal with the problem that the small-sample properties of the simple t- and F-statistics are such that standard critical levels may not provide a reliable guide to inference, we designed a bootstrap experiment. This procedure repeatedly draws values of the forecast errors (e1, ...., eT) with replacement from the empirical distribution function to construct a sample whose length (T) is identical to that of the original data sample. Having constructed an artificial sample in this way (e1(b)b,.,eT(b)b), where b is an indicator for the bth bootstrap and 1(b), …, T(b) are randomly drawn integer values between 1 and T, we recalculate the test statistics of interest, e.g., t− and F −statistics associated with the efficiency regressions. We repeat this in 5,000 bootstrap experiments to construct a histogram for the distribution of the test statistic. The value of the test statistic found for the actual data is then compared with this bootstrapped distribution to get bootstrapped p-values. We shall report the proportion of countries for which the actual test statistic exceeds the 95th percentile of the bootstrapped distribution (using a two-sided test for the t-statistic).

Using equation (1) the first two columns of Table 4 report the proportion of included countries in the various regions for which the t-statistic associated with the mean forecast error is less than -2 or greater than 2, respectively. The third column reports the proportion of bootstrapped p-values for α = 0 that fall below 0.05 using a two-sided test. The fourth column reports the percentage of regressions for which the absolute value of the t-statistic of β in the weak efficiency regression (2) is greater than two. The fifth column reports the percentage of cases where the F-test for the joint hypothesis α = 0, β = 0 in (2) exceeds its 5 percent critical level, while the final column reports the percentage of significant values of a sign test for whether the proportion of positive forecast errors differs from one-half, again using a 5 percent critical level. The purpose of reporting so many test statistics is to get a broader picture of possible forecast inefficiencies and to account for the fact that the individual test statistics are surrounded by more than the usual uncertainty due to the very small samples entertained here. Caution should therefore be exercised when interpreting the results.

Table 4.Results for Program Countries
Number of ObservationsMean ErrorFraction of Negative Errors
GDP forecasts
April current-year forecast errors958-0.850.53
September current-year forecast errors958-0.290.43
April next-year forecast errors902-1.510.60
September next-year forecast errors902-1.370.59
Inflation
April current-year forecast errors9272.110.41
September current-year forecast errors9430.920.47
April next-year forecast errors8484.510.36
September next-year forecast errors8494.120.38
Source: Author’s calculations.
Source: Author’s calculations.

A. GDP Forecasts

First consider the April current-year forecasts. For close to 40 percent of the countries in the African region, the GDP growth forecasts were systematically too large.6 The bootstrapped test statistics confirm a significant bias for a much larger proportion of African countries—close to 30 percent—than should be expected if the forecasts were genuinely unbiased. This proportion is reduced to 15 percent when bias and serial correlation are jointly tested, most likely because of the weaker power of the joint test which requires estimation of an additional parameter. In fact we can only identify significant serial correlation for about 6 percent of the African countries (column four). Similarly, for around 15 percent of the African countries, the proportion of positive signs in the current year forecast errors is significantly different from one-half at the 5 percent critical level.

Between 10 and 20 percent of the countries in CIS and Mongolia and the Western Hemisphere also show evidence of a significant bias in the forecasts. The region where serial correlation in the forecast errors appears to be most important is the Middle East where 15 percent of the countries generate significant bootstrapped test statistics.

These findings mostly carry over to the September current-year forecasts. Forecast errors continue to be biased and serially correlated for around 15 percent of the countries in Africa and there is strong evidence of serial correlation for the Middle East and Africa. In contrast there is very little evidence that the current-year forecasts are biased or serially correlated in developing Asia or the developing Economies. Overall, the proportion of cases with a significant bias is lower in the September current-year forecasts compared to the April current-year forecasts.

Turning to the next year forecast errors, there is evidence of a significant upward bias in the forecasts for around 35 percent of the countries in Africa and almost 30 percent of the countries in the Western Hemisphere. Significant biases also affect more than 20 percent of the countries among the Advanced Economies. Serial correlation in next-year forecast errors plagues all regions, particularly Africa. All told, the bootstrapped p-values show a pattern of biased or serially correlated next-year forecast errors in all regions.

Current-year forecast revisions are biased for Africa and the Western Hemisphere but there is little evidence of serial correlation. Next-year forecast revisions are biased and serially correlated for more than 10 percent of the countries in the Western Hemisphere but otherwise the evidence against (weak) efficiency tends to be relatively mild.

Effect of IMF programs

A potential source of bias in the WEO forecasts is whether or not a country is engaged in one of the IMF programs. The bias could, as discussed by Mussa and Phillips (2002) work in either direction. Given the limited sample size available here, we consider the effect of any of the following types of programs, namely SBA (Stand By Arrangements), EFF (Extended Fund Facility), SAF (Structural Adjustment Facility), ESAF (Enhanced Structural Adjustment Facility) and PRGF (Power Reduction and Growth Facility). Our data comprises 350 program events, some of which lasted multiple years and were still continuing at the end of the sample. Discarding extreme outliers we ended up with between 850 and 960 program year observations to analyze.

The results, shown in Table 5, indicate the presence of systematic overpredictions (negative mean forecast errors) of GDP growth in program countries. The upward bias is smallest for the September current year forecasts and largest for the April next year forecast. A full 60 percent of the next year forecast errors are negative for the program countries.

Table 5.Predictability of Forecast Errors in Relation to Current Information Variables(Fraction of all countries in region with t-values for additional variables above or below indicated threshold)
US GDP GrowthGerman GDP

Growth
Oil PricesGlobal

Current Account

Discrepancy
<-2>2<-2>2<-2>2<-2>2
Real GDP
April current-year forecast errors
Africa0.020.040.000.060.040.040.060.06
Central/Eastern Europe0.000.330.000.070.000.130.000.20
CIS and Mongolia0.000.000.080.230.080.080.230.00
Developing Asia0.000.050.000.000.000.000.050.10
Middle East0.080.080.000.000.000.000.000.00
Western Hemisphere0.000.090.060.000.060.060.030.09
Advanced economies0.000.310.070.030.070.000.000.38
September current-year forecast errors
Africa0.000.050.050.070.000.120.070.02
Central/Eastern Europe0.000.080.000.000.000.150.080.15
CIS and Mongolia0.000.000.000.080.000.250.330.08
Developing Asia0.000.090.180.000.000.000.090.00
Middle East0.000.000.100.000.100.000.000.00
Western Hemisphere0.000.000.050.000.000.000.050.00
Advanced economies0.030.240.070.070.000.030.000.14
April next-year forecast errors
Africa0.000.000.020.020.020.130.060.02
Central/Eastern Europe0.000.000.070.000.130.070.000.07
CIS and Mongolia0.000.080.080.540.000.080.000.00
Developing Asia0.000.090.040.040.000.040.040.04
Middle East0.000.070.000.070.070.000.070.00
Western Hemisphere0.030.060.090.060.060.000.000.09
Advanced economies0.210.030.030.000.210.000.000.24
September next-year forecast errors
Africa0.020.000.020.040.020.060.060.06
Central/Eastern Europe0.000.000.070.000.070.000.070.00
CIS and Mongolia0.000.000.000.380.000.000.000.00
Developing Asia0.090.040.040.040.000.000.000.04
Middle East0.070.070.000.070.070.000.070.00
Western Hemisphere0.060.000.060.150.150.000.030.12
Advanced economies0.070.000.100.000.210.000.000.03
Current-year forecast revision
Africa0.050.050.050.020.070.000.050.05
Central/Eastern Europe0.000.360.000.090.000.000.000.00
CIS and Mongolia0.000.000.100.500.000.200.000.00
Developing Asia0.000.150.080.000.000.000.080.00
Middle East0.100.100.000.000.000.000.200.00
Western Hemisphere0.000.070.070.000.000.070.130.13
Advanced economies0.000.290.040.040.040.000.000.32
Next-year forecast revision
Africa0.070.020.020.070.020.020.020.05
Central/Eastern Europe0.000.000.110.000.000.110.220.00
CIS and Mongolia0.000.000.000.140.000.140.000.00
Developing Asia0.000.000.000.080.000.000.170.00
Middle East0.000.000.000.000.000.000.500.00
Western Hemisphere0.070.000.070.070.200.000.000.00
Advanced economies0.040.000.000.040.040.000.000.11
Inflation
April current-year forecast errors
Africa0.100.080.000.150.020.020.060.17
Central/Eastern Europe0.000.200.070.400.000.000.130.00
CIS and Mongolia0.000.080.000.690.000.000.150.15
Developing Asia0.000.050.050.300.000.050.000.00
Middle East0.150.000.150.000.000.000.000.15
Western Hemisphere0.030.250.000.220.060.030.030.03
Advanced economies0.000.140.070.030.000.070.210.03
September current-year forecast errors
Africa0.020.000.020.120.000.080.020.14
Central/Eastern Europe0.000.070.070.270.070.070.070.13
CIS and Mongolia0.000.000.080.620.000.000.540.08
Developing Asia0.000.080.080.080.080.040.040.04
Middle East0.070.070.210.000.070.000.000.07
Western Hemisphere0.000.180.000.180.000.000.060.00
Advanced economies0.030.030.000.100.000.140.070.00
April next-year forecast errors
Africa0.000.080.000.150.060.060.040.06
Central/Eastern Europe0.000.330.000.330.000.000.000.00
CIS and Mongolia0.000.230.000.920.000.000.000.00
Developing Asia0.130.300.170.260.000.040.090.09
Middle East0.140.140.070.000.000.000.000.07
Western Hemisphere0.000.300.000.270.060.120.090.06
Advanced economies0.070.170.240.140.030.000.000.00
September next-year forecast errors
Africa0.000.120.020.120.060.060.020.06
Central/Eastern Europe0.000.330.000.470.070.000.000.00
CIS and Mongolia0.000.230.000.770.000.000.080.08
Developing Asia0.080.250.170.250.000.040.130.00
Middle East0.070.070.140.000.000.000.000.07
Western Hemisphere0.000.270.000.270.060.060.150.06
Advanced economies0.070.170.210.100.000.030.000.00
Current-year forecast errors
Africa0.040.080.000.080.040.080.040.02
Central/Eastern Europe0.000.130.130.270.000.000.130.07
CIS and Mongolia0.000.080.000.540.000.000.000.31
Developing Asia0.000.080.000.040.130.080.000.04
Middle East0.070.070.070.070.000.070.000.00
Western Hemisphere0.000.060.000.060.030.090.000.03
Advanced economies0.000.100.100.070.000.030.070.03
Next-year forecast revision
Africa0.000.040.060.040.020.120.020.02
Central/Eastern Europe0.000.130.000.000.000.000.070.00
CIS and Mongolia0.000.150.000.230.000.000.000.23
Developing Asia0.000.040.090.040.040.040.000.04
Middle East0.000.000.070.140.000.210.070.00
Western Hemisphere0.000.000.150.030.060.120.000.03
Advanced economies0.000.070.000.100.000.030.000.00
Current Account (in percent of GDP)
April current-year forecast errors
Africa0.040.000.000.020.000.060.060.02
Central/Eastern Europe0.000.070.000.000.000.000.070.20
CIS and Mongolia0.000.000.000.000.000.000.310.08
Developing Asia0.160.000.000.050.000.050.050.00
Middle East0.090.000.000.090.000.090.090.00
Western Hemisphere0.060.000.000.060.060.060.030.06
Advanced economies0.040.040.110.040.070.040.180.00
September current-year forecast revision
Africa0.090.020.020.000.000.120.120.00
Central/Eastern Europe0.070.000.000.000.000.000.290.14
CIS and Mongolia0.080.000.000.080.000.080.230.23
Developing Asia0.090.090.000.000.000.000.090.00
Middle East0.130.250.000.250.130.000.130.00
Western Hemisphere0.000.180.060.120.000.060.000.00
Advanced economies0.070.040.070.040.110.070.110.00
April next-year forecast errors
Africa0.110.000.020.000.020.110.110.04
Central/Eastern Europe0.000.130.070.070.000.070.070.13
CIS and Mongolia0.150.000.000.080.080.080.080.08
Developing Asia0.090.000.000.000.000.170.130.00
Middle East0.070.140.000.210.070.000.000.07
Western Hemisphere0.090.060.030.090.000.090.000.06
Advanced economies0.070.140.110.070.000.140.140.07
September next-year forecast errors
Africa0.020.000.020.020.040.130.220.04
Central/Eastern Europe0.000.070.070.070.000.070.070.07
CIS and Mongolia0.380.000.000.000.150.000.000.15
Developing Asia0.040.040.040.090.000.170.130.00
Middle East0.000.210.070.070.140.000.000.07
Western Hemisphere0.000.090.000.090.030.090.030.06
Advanced economies0.110.070.140.070.000.070.110.04
Current-year forecast errors
Africa0.020.000.020.020.040.110.020.09
Central/Eastern Europe0.000.000.000.000.000.000.070.13
CIS and Mongolia0.080.000.000.080.000.000.170.08
Developing Asia0.000.000.000.000.000.000.080.08
Middle East0.000.000.000.000.090.000.000.00
Western Hemisphere0.130.000.000.130.000.130.060.06
Advanced economies0.110.210.070.000.070.000.000.00
Next-year forecast errors
Africa0.000.020.020.020.040.020.020.04
Central/Eastern Europe0.070.000.000.070.000.000.070.13
CIS and Mongolia0.000.000.000.000.000.000.000.08
Developing Asia0.000.000.000.000.000.000.000.00
Middle East0.090.000.090.000.000.090.000.00
Western Hemisphere0.190.000.000.130.000.190.000.06
Advanced economies0.070.040.070.040.070.040.040.00
Source: Author’s calculations.
Source: Author’s calculations.

Although the bias estimates appear large, it should be borne in mind that so were the average biases reported for countries in the same regions hosting most of the program countries (see Table 3).

B. Inflation Forecasts

As mentioned previously, the inflation data is affected by numerous outliers so we will not rely on standard test statistics and instead move directly to consider the bootstrap results. These reveal mild evidence of inefficiency in the current-year inflation forecasts. There appears to be some positive bias (underprediction of inflation) in the case of Africa and the Western Hemisphere. By far the strongest evidence against efficiency is found in the next-year forecast errors which reveal forecasts that are systematically downward-biased in most regions except for the Advanced Economies. However, forecast errors are serially correlated in the latter region so the null of no bias or serial correlation is rejected for around 15 percent of all countries (more than double the level expected under the null).

For the next year forecasts, with the exception of CIS and Mongolia, a greater-than-expected proportion of countries in the various regions generate a significant test statistic associated with the bias. The strongest evidence against efficiency comes from the serial correlation tests in column five, however, which shows that p-values below 5 percent were generated for between 15 percent and 40 percent of the countries in the various regions. In particular, more than 30 percent of the countries in the Western hemisphere show evidence of significant serial correlation in the forecast errors.

With few exceptions, forecast revisions reveal little systematic evidence of biases or serial correlation.

Effect of IMF programs

For the inflation forecasts again a large and systematic bias is observed for the countries enrolled in Fund-sponsored projects. However, the results in Table 5 show that the bias now goes in the opposite direction relative to what was observed for GDP growth as the inflation rate is underpredicted and the mean forecast error therefore takes a positive value. Again the largest bias—equal to 4.5 percent—is observed at the longest forecast horizon, i.e. for the April next year forecasts, while the mean bias is a more modest 0.9 percent for the September current year predictions.

C. Import and Export Volume, Current Account Balance

There is little evidence of systematic inefficiencies in the current-year or next year forecasts of trading volume. A similar conclusion holds for the current-year forecasts of the current account balance. However there does seem to be some evidence of bias and serial correlation in the next-year forecasts of the current account balance, where rejection rates are twice their expected values for many of the regions and sometimes four times as high (e.g., 24 percent in the case of the April next year forecasts for the Western Hemisphere or Developing Asia), often as a result of serial correlation.

VI. Can The WEO Forecast Errors Be Predicted?

The process whereby the WEO forecasts are generated puts considerable emphasis on integrating predictions across countries, regions and variables in order to produce a coherent and internally consistent projection of current and future economic activity. One way to analyze whether the procedures that are currently in place have their intended effect is to test for informational efficiency using a range of indicators of global economic activity. Such tests build on the moment condition E[et+1|Ωt] = 0, where Ωt is the forecaster’s information set at the time of the forecast (t), and are hence versions of the efficiency tests in (4).

In our empirical application we focus on four such predictor variables. First, we consider the WEO prediction of U.S. GDP growth. This is an obvious choice given the size and leading role that the U.S. economy plays in shaping global economic activity. The second instrument is the WEO prediction of German output growth—again motivated by the significance of this economy to regional and global growth.7 Finally, we also use the WEO forecast of oil prices and a global current account discrepancy instrument as predictors. Oil prices are an obvious choice since they are an important determinant of economic growth and inflation in a number of economies. The global current account discrepancy is constructed as the sum total of current accounts across all countries scaled by 15 global export. This figure should be equal to zero but may differ from this value due to measurement errors.

Table 6 shows the outcome of this exercise. Within each region and for each of the predictor variables the table reports the proportion of t-values below minus two and above two, respectively. Results indicative of a failure to fully account for the predicted U.S. GDP growth should show up in the form of a proportion of significant t-values somewhat higher than 5 percent. There is also information in the sign of the t-statistic. For GDP growth a higher proportion of positive and significant values than negative and significant t-statistics would reveal a failure to fully account for the spillover of U.S. GDP growth to other countries.

Table 6.Output Gaps and the Predictability of Forecast Errors in Advanced Economies(Value of t-statistics for the coefficient of the output gap in forecast efficiency regression)
Current-YearNext-YearForecast Revisions
AprilSeptemberAprilSeptemberCurrent -

Year
Next-

Year
Real GDP
Australia-1.640.91-0.030.91-2.69-1.91
Austria-1.56-1.41-1.09-1.34-1.10-0.24
Belgium-2.01-1.51-1.35-1.54-0.730.13
Canada-0.58-0.530.53-0.22-0.28-0.04
Cyprus0.020.020.000.00-0.440.76
Denmark-1.67-0.25-1.20-0.85-3.02-2.11
Finland-0.17-0.04-0.30-0.09-0.420.25
France-2.32-1.77-1.73-2.42-1.53-0.87
Germany-2.590.11-4.44-2.19-2.43-1.34
Greece1.790.620.17-0.29-0.25-0.20
Hong Kong SAR-1.750.01-2.30-4.08-1.23-0.99
Iceland-1.72-1.13-0.96-0.62-0.750.18
Ireland-0.310.84-0.36-0.93-0.69-0.68
Israel-1.46-0.98-1.43-1.57-0.42-0.26
Italy-1.97-4.05-3.14-2.30-1.38-1.32
Japan-0.311.40-2.63-0.62-0.17-0.47
Korea-1.19-1.74-1.98-2.32-0.370.69
Luxembourg1.140.701.110.93-0.080.69
Netherlands-0.14-0.05-1.26-0.36-0.320.63
New Zealand1.21-0.67-0.55-0.471.210.53
Norway-1.45-2.990.530.21-0.370.32
Portugal0.61-0.240.680.11-0.420.96
Singapore-0.75-0.69-3.08-2.67-0.330.48
Spain-0.79-1.020.190.14-0.54-0.14
Sweden-2.03-0.38-1.33-2.63-1.65-1.73
Switzerland0.542.38-0.69-0.020.280.54
Taiwan, Province of China-0.730.55-0.83-1.26-0.85-0.42
United Kingdom-1.57-0.67-0.21-0.82-1.76-0.42
United States-0.32-1.020.62-0.070.951.21
Inflation
Australia-0.31-0.270.32-0.06-0.090.23
Austria-0.35-0.280.060.86-0.28-0.32
Belgium1.131.841.340.710.930.25
Canada0.680.740.992.341.220.82
Cyprus-0.90-4.180.000.00-0.71-0.21
Denmark0.29-1.080.59-0.390.420.00
Finland2.262.170.771.811.60-0.37
France1.801.341.692.911.140.44
Germany1.071.710.651.421.11-0.12
Greece0.070.081.420.42-0.16-0.30
Hong Kong SAR-0.070.01-0.93-1.000.07-0.81
Iceland0.721.601.550.971.210.51
Ireland1.230.672.311.481.521.38
Israel1.770.411.530.401.63-1.23
Italy0.11-1.31-1.29-0.520.59-0.99
Japan1.54-1.09-0.13-0.522.64-0.06
Korea1.583.151.332.082.07-0.78
Luxembourg-0.190.26-0.280.15-0.06-0.55
Netherlands1.861.213.163.391.01-0.27
New Zealand1.140.220.530.190.81-0.54
Norway0.05-0.64-1.18-2.090.930.51
Portugal1.891.242.161.110.570.85
Singapore-0.37-0.74-0.570.08-0.201.30
Spain-0.12-0.45-0.30-0.180.70-0.12
Sweden0.64-0.251.300.340.470.54
Switzerland1.19-0.570.651.781.40-0.56
Taiwan, Province of China-2.19-1.45-0.92-0.53-1.47-0.09
United Kingdom1.240.15-0.40-0.212.27-0.09
United States-0.12-0.180.990.11-0.260.99
Current Account
Australia0.400.620.610.950.200.89
Austria0.34-1.370.77-0.270.780.47
Belgium-1.240.33-1.540.96-2.03-2.09
Canada2.600.851.070.952.942.26
Cyprus1.02-2.740.000.001.454.27
Denmark0.330.371.390.07-0.20-0.09
Finland0.070.14-0.750.94-1.25-1.45
France-0.57-1.40-1.79-5.10-0.170.10
Germany1.87-0.151.690.231.350.48
Greece-0.390.18-0.540.29-0.17-0.20
Hong Kong SAR-1.02-2.160.15-4.350.200.12
Iceland-0.610.44-0.98-0.58-1.85-1.83
Ireland0.530.870.780.31-0.29-0.10
Israel0.47-0.881.632.110.511.65
Italy2.541.280.790.390.22-0.48
Japan3.251.263.822.221.270.01
Korea-0.330.170.660.63-0.39-0.16
Luxembourg-0.33-0.24-0.800.31-1.79-1.73
Netherlands-1.17-1.75-3.00-2.490.080.11
New Zealand-1.69-0.56-1.310.171.07-1.25
Norway-0.19-1.46-0.53-0.890.010.27
Portugal-2.85-0.77-2.23-1.47-0.92-1.08
Singapore-5.52-1.17-1.44-0.67-1.95-3.66
Spain-1.57-1.78-1.84-2.67-0.51-0.59
Sweden1.912.781.931.940.26-0.47
Switzerland-0.311.49-1.060.54-3.47-3.16
Taiwan, Province of China-0.93-3.28-0.17-1.381.450.13
United Kingdom-2.10-0.91-0.73-0.83-1.27-0.57
United States0.09-0.18-0.150.400.870.00
Source: Author’s calculations.
Source: Author’s calculations.

There are only few cases where the WEO prediction of U.S. GDP growth appears to be correlated with the forecast errors. However, the ones that we find are of considerable interest. Indeed the evidence suggests that, for the Advanced Economies, 31 percent of the April current year forecasts and 24 percent of the September current year U.S. GDP forecasts generate a t-value above two and hence predict the forecast errors. This leads to a significantly positive t-statistic for 29 percent of the current-year forecast revisions in this region. In contrast there is no evidence that the U.S. GDP forecast has predictive power over the next-year forecast errors. The only other instance registering a greater than expected proportion of significant t-values is the current year forecasts for central and eastern Europe, where 33 percent of the t-values exceed a value of two. For many of the countries in this region, the revision to the current year forecast that takes place between April and September is predicted by the U.S. GDP forecast.

Turning to the WEO forecast of German output growth, interestingly this is positively correlated and significant in explaining forecast errors in a high proportion of countries in CIS and Mongolia (particularly for the next year forecast errors) but not to nearly the same extent in other regions.

With a few interesting exceptions—namely CIS and Mongolia for which predicted oil prices are positively correlated with forecast errors in GDP growth and Western Hemisphere and Advanced Economies for which a negative correlation emerges—the WEO forecasts of oil prices do not appear to be overly important in explaining forecast errors in output growth.

Interestingly, the global current account discrepancy is significant for 40 percent and 24 percent of the Advanced Economies in explaining the April current year and next year forecast errors, respectively.

There is evidence that the next year inflation forecast errors are linked to U.S. GDP forecasts, particularly for countries in central and eastern Europe, CIS, and Mongolia, Developing Asia, Western Hemisphere, and the Advanced Economies. Once again the WEO forecast of German output growth is significant in explaining the inflation forecast error for a very large proportion of the countries in the CIS and Mongolia region.

Both the WEO prediction of U.S. growth and oil prices affect the error in forecasting the current account for 20–30 percent of the countries in the Middle East. Furthermore, the global current account discrepancy affects the current account forecast errors in a greater than expected number of cases.

A. Output Gap

The output gap—measured as the difference between actual and potential GDP—plays an important role in the WEO forecasts. Implicit in these is an assumption that the output gap is eliminated after five years. If this assumption is unrealistic and leads to biased forecasts, then one would expect that the predicted value of the output gap itself be accountable for forecast errors. For example, if it takes longer to eliminate the output gap than assumed in the WEO, then the WEO will tend to overpredict forecasts for countries with large output gaps.

We have data on output gaps for the 29 Advanced Economies. For each of these we regress the forecast error on an intercept and the predicted output gap whose timing corresponds to the forecast with which it gets matched.

Table 7 presents the results in the form of t-statistics for current and next year forecast errors and forecast revisions. A pattern that stands out for the GDP forecasts is that the signs of the estimated t-values predominantly are negative. Around 15 percent of the t-statistics exceed two in absolute value. The large negative t-statistics for Germany, France, and Italy are particularly interesting since, as we shall see subsequently, these were also economies for which the WEO output growth forecasts were systematically upwards biased during the period. This finding suggests that the reduction in the output gap assumed in computing the WEO forecasts could lead to overpredictions: All three economies had large output gaps during the 1990s as did Japan—the output gap averaged -1.63, -1.99, -2.30, and -4.16 for France, Germany, Italy, and Japan, respectively. These were among the highest output gaps in the 29 countries. An assumption in the WEO forecasts that these output gaps would be reduced too fast might lead to a greater prediction of output growth and hence to an upward bias in the forecast.

Table 7.Efficiency Scores, by Country1/(Zero score best, three worst)
GDP GrowthInflationCurrent Account
Current-

Year
Next-

Year
Current-

Year
Next-

Year
Current-

Year
Next-

Year
Africa
Algeria131200
Benin001001
Botswana001001
Burkina Faso000000
Burundi211201
Cameroon011000
Cape Verde110011
Central African Republic210010
Chad000000
Comoros131002
Congo, Democratic Republic221011
Congo, Republic of121000
Cote d’Ivoire121001
Djibouti110001
Equatorial Guinea010011
Ethiopia100000
Gabon000000
Gambia, The001211
Ghana111111
Guinea110000
Guinea-Bissau001300
Kenya111311
Lesotho000111
Madagascar011100
Malawi112200
Mauritius101000
Morocco220011
Mozambique002210
Namibia111110
Niger001010
Nigeria001300
Rwanda001000
Sao Tome212310
Senegal101001
Seychelles100111
Sierra Leone221100
South Africa211000
Sudan022100
Swaziland001000
Tanzania122300
Togo221001
Tunisia000001
Uganda100010
Zambia102202
Zimbabwe110210
Regional average0.760.800.780.780.350.46
Central and Eastern Europe
Albania000010
Bulgaria210012
Croatia001000
Czech Republic011112
Estonia002200
Hungary110200
Latvia102201
Lithuania112200
Macedonia211000
Malta010001
Poland112200
Romania221301
Slovak Republic210011
Slovenia113211
Turkey000010
Regional average0.870.7311.070.400.60
CIS and Mongolia
Armenia111211
Azerbaijan121210
Belarus122200
Georgia111200
Kazakhstan122201
Kyrgyz Republic012111
Moldova212200
Mongolia021301
Russia002311
Tajikistan122321
Ukraine012212
Uzbekistan222201
Regional average0.831.421.672.170.580.75
Developing Asia
Bangladesh000000
China131100
India101212
Indonesia000021
Malaysia001100
Pakistan110212
Papua New Guinea001201
Philippines111100
Sri Lanka000110
Thailand000010
Vanuatu110100
Vietnam011200
Regional average0.420.580.501.080.500.50
Middle East
Bahrain120301
Egypt112101
Iran101100
Jordan100032
Kuwait101102
Lebanon022310
Oman100100
Qatar001100
Saudi Arabia001202
Syrian Arab Republic011001
United Arab Emirates000000
Regional average0.550.550.821.180.360.82
Western Hemisphere
Argentina002001
Bolivia131011
Brazil102222
Chile000000
Colombia222202
Costa Rica000102
Dominican Republic110000
Ecuador100300
El Salvador021200
Guatemala213301
Mexico013210
Panama001100
Peru102200
Trinidad and Tobago001300
Uruguay002311
Venezuela000311
Regional average0.560.631.251.690.380.69
Advanced Economies
Australia101210
Austria001011
Belgium121001
Canada000111
Cyprus100003
Denmark101111
Finland122111
France110012
Germany120001
Greece112200
Hong Kong SAR001200
Iceland200001
Ireland120102
Israel111010
Italy120001
Japan010000
Korea000000
Netherlands120002
New Zealand110111
Norway000100
Portugal220002
Singapore110112
Spain011111
Sweden020112
Switzerland112122
Taiwan, Province of China001203
United Kingdom110001
United States000010
Regional average0.710.890.500.640.501.11

Aggregate score of the indicator variables for the three efficiency tests—tests for bias, serial correlation, and predictability by means of any of the four predictor variables examined earlier (Table 5). An indicator value of 1 indicates that the test was failed (rejection of the null hypothesis of an efficient forecast).

Aggregate score of the indicator variables for the three efficiency tests—tests for bias, serial correlation, and predictability by means of any of the four predictor variables examined earlier (Table 5). An indicator value of 1 indicates that the test was failed (rejection of the null hypothesis of an efficient forecast).

The sign of the regression coefficient of the output gap is predominantly positive in the case of the inflation forecast errors, i.e. the opposite sign of what was found for the GDP forecasts. Hence the WEO tends to underpredict inflation, the larger the output gap, i.e. the greater an economy’s unused capacity. This effect can be quite large and is borderline significant for countries such as France, Germany, and Korea.

Finally, turning to the regression results for the current account, there are many instances with large and significant predictability from the output gap over subsequent forecast errors, although the sign of the regression coefficient varies quite a bit. Countries for which a significant degree of predictability is found include Hong Kong, Japan, the Netherlands, Singapore, and Sweden.

B. Overall Scores, by Country and Region

So far we have tested the efficiency properties of the growth, inflation and current account forecasts and reported results aggregated by region. It is also of interest to study forecast efficiency at the level of the individual countries. Indications of problems for a particular country model may suggest that the model should be modified. For this purpose, we create an aggregate score. Each of three efficiency tests—tests for bias, serial correlation, and predictability by means of any of the four predictor variables examined—is associated with an indicator variable that is one if the relevant t-statistic exceeds two in absolute value. Otherwise the indicator variable equals zero. This means that each country gets a score of zero (no detectable problem) and three (detectable problems along all dimensions). We produce these scores for current year and next year forecasts of GDP growth, inflation, and the current account.

Results for individual countries along with regional averages are shown in Table 8. Interestingly, it is not the case that the Advanced Economies is the region in which the average score is lowest. This may have to do with the greater importance of measurement errors and outliers in some of the other regions. It can be easier to detect relatively small inefficiencies in economies not affected by outliers.

Table 8.Sign Tests for Directional Accuracy(Average across regions except for last column)
Fraction of Observations

with Correct Sign
Benchmark

Estimates
PT Statistic1/Fraction of Significant

PT Test Statistics2/
Real GDP
April current-year forecast errors
Africa0.730.521.810.46
Central and Eastern Europe0.800.532.130.67
CIS and Mongolia0.670.521.160.23
Developing Asia0.780.522.120.65
Middle East0.770.512.060.62
Western Hemisphere0.710.511.610.38
Advanced economies0.790.502.280.66
September next-year forecast errors
Africa0.660.501.400.24
Central and Eastern Europe0.700.531.260.36
CIS and Mongolia0.610.550.550.08
Developing Asia0.680.511.350.26
Middle East0.710.511.550.29
Western Hemisphere0.650.501.260.18
Advanced economies0.620.481.190.14
Export Volume
April current-year forecast errors
Africa0.750.511.880.57
Central and Eastern Europe0.760.521.910.36
CIS and Mongolia0.810.512.100.45
Developing Asia0.740.501.880.63
Middle East0.850.502.610.79
Western Hemisphere0.790.502.260.71
Advanced economies0.760.492.110.61
September next-year forecast errors
Africa0.700.491.620.40
Central and Eastern Europe0.730.501.730.50
CIS and Mongolia0.750.511.660.46
Developing Asia0.740.511.830.58
Middle East0.710.501.570.29
Western Hemisphere0.720.501.710.45
Advanced economies0.690.501.540.36
Import Volume
April current-year forecast errors
Africa0.740.511.840.47
Central and Eastern Europe0.790.502.150.73
CIS and Mongolia0.720.501.530.36
Developing Asia0.740.501.900.53
Middle East0.810.512.290.71
Western Hemisphere0.810.512.370.77
Advanced economies0.770.502.160.64
September next-year forecast errors
Africa0.730.501.830.46
Central and Eastern Europe0.780.512.020.60
CIS and Mongolia0.730.511.560.55
Developing Asia0.740.511.830.41
Middle East0.770.521.990.43
Western Hemisphere0.770.512.070.64
Advanced economies0.700.491.650.46
Inflation
April current-year forecast errors
Africa0.740.521.830.44
Central and Eastern Europe0.820.622.080.73
CIS and Mongolia0.780.621.650.54
Developing Asia0.690.511.470.45
Middle East0.680.501.370.23
Western Hemisphere0.730.531.760.34
Advanced economies0.760.531.890.55
September next-year forecast errors
Africa0.690.521.460.24
Central and Eastern Europe0.700.590.920.21
CIS and Mongolia0.790.611.650.36
Developing Asia0.630.511.080.21
Middle East0.620.491.060.14
Western Hemisphere0.680.531.440.40
Advanced economies0.670.511.360.24
Current Account (in percent of GDP)
April current-year forecast errors
Africa0.600.510.700.11
Central and Eastern Europe0.710.501.540.29
CIS and Mongolia0.670.511.170.08
Developing Asia0.700.501.620.44
Middle East0.690.531.290.45
Western Hemisphere0.650.501.190.20
Advanced economies0.630.510.930.14
September next-year forecast errors
Africa0.590.500.730.13
Central and Eastern Europe0.640.491.090.20
CIS and Mongolia0.620.491.020.17
Developing Asia0.650.501.160.30
Middle East0.660.491.300.14
Western Hemisphere0.600.500.750.13
Advanced economies0.540.510.240.07
Source: Author’s calculations.

The PT test is a sign that proposed by Pesaran and Timmermann (1992).

For all countries included in a region.

Source: Author’s calculations.

The PT test is a sign that proposed by Pesaran and Timmermann (1992).

For all countries included in a region.

With the exception of the Advanced Economies, the inflation forecasts generate the highest “problem scores.” These are on average above one in most regions, suggesting that at least one of the efficiency tests is rejected. Furthermore, for most regions the next-year forecasts generate higher average scores than the current-year forecasts, so the greatest inefficiencies are to be found in the next year forecasts.

VII. Directional Accuracy

The anticipation of turning points is acknowledged to be one of the most difficult exercises in economic forecasting. If done with any measure of success it is also one of the potentially most rewarding ones. Most obviously, the ability to foresee a recession will allow policy makers to take actions and install remedies that will cushion the downturn in economic activity.

Defining turning points is made difficult by the fact that official definitions of recession periods (such as the NBER recession indicator available for the United States) are not widely published for the majority of countries included in our sample. Furthermore, recession periods do not always coincide with calendar years and our forecasts are only available on an annual basis. To deal with these problems we simply study the extent to which the WEO forecasts predicted the sign of the variables of interest. Although this measure is subject to the criticism that it need not accurately track recessions, it has the advantage that it is more robust to the type of outliers that are so prevalent in many of the regions outside the Advanced Economies.

The test statistic underlying our analysis is the sign test proposed by Pesaran and Timmermann (1992) which takes the form:

where P^ is a sample estimate of the proportion of positive signs that are correctly predicted, P^* is an estimate of the probability of correctly predicting the signs under the assumption that predictions and realizations are independently distributed, and V^(P^), V^(P^*) are consistent estimates of the variances of P^ and P^*, respectively. P^* sets a natural benchmark for sign predictions for a forecaster with no genuine prediction skills. It can be consistently estimated by P^*=P^YP^Y^+(1P^Y(1P^y¯), where P^y is the estimated proportion of positive values of the target (outcome) variable while P^y¯ is the estimated proportion of positive values of the predictions.

For example, if the proportion of positive realizations is 80 percent, then a forecaster without any skill that predicts positive growth, say, 90 percent of the time by flipping a biased coin can expect to correctly predict 74 percent of the signs.8 The test statistic corrects for this by benchmarking forecasts relative to P^*.

Results from applying this test are shown in Table 9. The table shows the outcomes averaged within each of the seven regions and categorized by the time of the prediction. The first column shows the percentage of correctly predicted signs followed by a column of benchmark estimates, P^*. If the forecasts contain predictive sign information we should expect the first column to exceed the second column and the higher the differential, the more precise the sign information embedded in the forecast. Columns three and four show the average of the PT test statistic in addition to the percentage of PT statistics across each region that exceeds two.

Table 9.Real GDP: Significance of Forecast Revisions After Executive Board Meeting(Average across regions except for fractions)
Fractions of
T-Valuesβ^
tβ^<2tβ^>2Coefficients>0R-SquareMSE-Ratio
April current-year forecast errors
Africa0.000.130.810.110.83
Central and Eastern Europe0.000.070.600.090.85
CIS and Mongolia0.000.150.540.070.48
Developing Asia0.000.150.500.160.72
Middle East0.000.000.620.030.77
Western Hemisphere0.000.130.530.080.64
Advanced economies0.000.520.900.230.81
September current-year forecast errors
Africa0.000.070.410.070.55
Central and Eastern Europe0.000.230.690.140.69
CIS and Mongolia0.000.330.670.180.67
Developing Asia0.000.270.550.150.43
Middle East0.000.600.800.360.75
Western Hemisphere0.000.380.670.240.74
Advanced economies0.000.410.830.270.72
April next-year forecast errors
Africa0.000.100.630.080.92
Central and Eastern Europe0.130.000.270.140.77
CIS and Mongolia0.000.000.380.040.36
Developing Asia0.000.040.480.070.63
Middle East0.070.000.430.071.64
Western Hemisphere0.030.120.520.080.66
Advanced economies0.030.140.620.120.95
September next-year forecast errors
Africa0.040.040.380.060.81
Central and Eastern Europe0.000.070.600.080.63
CIS and Mongolia0.000.000.380.010.55
Developing Asia0.040.170.480.150.74
Middle East0.000.140.570.090.73
Western Hemisphere0.030.060.450.090.80
Advanced economies0.000.070.590.070.90
Source: Author’s calculations
Source: Author’s calculations

First consider the directional accuracy of the GDP forecasts. Moving from April next year forecasts through the September current year forecasts, as the information underlying the WEO forecasts becomes more accurate one should expect the percentage of correctly predicted forecasts to improve and the average PT sign test to go up. While this is generally found to be true, it is also clear that by far the largest improvement takes place in the revisions when moving from the September next year forecast to the April current-year forecast or from the April current-year forecast to the September current-year forecast. Conversely, there is little evidence that next-year forecast revisions occurring between April and September result in an improved ability to forecast the sign of future GDP growth.

The April and September next-year GDP growth forecasts do reveal some ability to forecast the direction of future economic activity. The benchmark percentage of correctly predicted signs (which embodies no information other than knowledge of the average, long-term percentage of years with positive growth) is around 50 percent while the proportion of predictions with correct signs lies between 62 and 71 percent. This gives rise to PT test statistics that are significant for about 20 percent of the countries.

Directional accuracy improves vastly when moving to the same-year forecasts. The proportion of correctly predicted signs now varies between 68 and 78 percent for the April forecasts and between 76 and 90 percent for the September forecasts. Furthermore, the average values of the PT test statistic and the percentage of these that are significant also go up by a large amount. For example, for the Advanced Economies the proportion of correctly predicted signs increases from 62 percent (next year forecasts) to 78 percent (April current-year forecasts) and 90 percent (September current-year forecasts). The “hit ratio”—i.e. the percentage of correctly predicted signs—obtains a high level of significance for all regions with exception of CIS and Mongolia.

Turning to the inflation forecasts, a similar pattern is observed: little improvement in the directional accuracy of the forecasts when moving from the April to the September next-year forecasts, followed by more significant improvements in the current-year forecasts. Interestingly, while the hit rate of the Advanced Economies’ next-year inflation forecasts is higher than was observed for the next-year GDP forecasts, the improvement when moving to the April current-year forecasts is smaller, so the directional accuracy of the September same-year forecasts is basically identical for the two sets of forecasts.

Finally, the directional accuracy of the current account forecasts is generally lower than that observed for inflation or output growth. As expected this gives rise to smaller average PT test statistics and a lower percentage of these that are significant at the 5 percent level.

VIII. Revisions from Board to Published Forecasts

WEO forecasts are published twice a year in April and September. Several rounds of forecast revisions precede the published version. A first set of predictions is presented to the IMF Board in February and July each year, preceding the April and September WEO publication. To assess the informational value of forecast revisions that occur between the Board version and the published version, we obtained data on Board forecasts of current-year GDP growth in February and next-year Board forecasts of GDP growth reported in July. We refer to these forecasts as y^t,tFeb and y^t+1,tJuly, respectively. Further, let the forecast revisions from the Board to the published WEO forecasts be given by revt,tpubBoard and revt+1,tpubBoard. If the revisions occurring between the Board and published forecasts contain useful information, we should expect that they help predict the errors in the original Board forecasts, defined as et,tBoard=yty^t,tFeb and et+1,tBoard=yt+1y^t+1,tJuly. We test this proposition through the regressions

If the revisions incorporated in the published WEO forecasts do not add any value to the original Board forecast then we should expect to find β−coefficients near zero. Conversely, we would expect to find significant and positive values of β and non-zero R2-values in case the revisions contain valuable information. Estimation results based on (10) are reported in Table 10. The current-year forecast errors for the Advanced Economies reveal strong evidence that the Board-to-Publication revision contains valuable information that not only is 20 significantly correlated with the forecast error for around 50 percent of the countries but has the required positive sign for between 80 and 90 percent of the countries. The large R2-value of around 0.25 is further testimony to this effect and suggests that 25 percent of the current-year February or July forecast error can be explained by the revision between the Board and published version.

Table 10.Description Statistics for Forecast Errors During 2001–2003 Recession and Recovery(Average across countries in region)
Forecast Error
Previous-YearCurrent-Year
OutcomeAprilSeptemberAprilSeptember
Real GDP
2001 recession results
Africa4.33-0.58-0.37-0.76-0.66
Central and Eastern Europe2.68-2.08-2.23-1.40-0.92
CIS and Mongolia7.873.933.312.922.74
Developing Asia2.63-2.08-2.19-1.52-1.03
Middle East3.520.420.00-0.34-0.61
Western Hemisphere1.14-2.97-2.90-2.62-1.98
Advanced economies1.53-2.05-2.17-1.32-0.35
2002 recovery results
Africa3.10-1.64-1.93-1.55-1.16
Central and Eastern Europe4.34-0.40-0.470.350.56
CIS and Mongolia6.251.431.751.551.06
Developing Asia3.36-1.32-1.22-0.47-0.38
Middle East2.39-1.38-1.580.210.03
Western Hemisphere0.14-3.88-3.64-0.310.26
Advanced economies1.89-1.44-0.92-0.010.11
2003 recovery results
Africa4.12-0.55-0.66-0.130.25
Central and Eastern Europe4.29-0.36-0.220.160.20
CIS and Mongolia8.914.264.003.583.43
Developing Asia4.980.240.330.510.50
Middle East5.261.471.521.151.23
Western Hemisphere2.75-0.96-0.011.181.62
Advanced economies1.76-1.59-1.11-0.160.40
Export Volume
2001 recession results
Africa6.38-3.31-2.43-2.01-1.60
Central and Eastern Europe5.46-3.94-2.41-2.89-2.48
CIS and Mongolia2.11-2.101.912.54-10.58
Developing Asia1.31-6.03-5.86-5.40-2.98
Middle East2.520.45-0.48-3.20-4.42
Western Hemisphere0.33-6.45-6.41-5.83-5.12
Advanced economies0.16-6.54-7.21-5.70-2.48
2002 recovery results
Africa4.89-1.1022.5823.02-2.53
Central and Eastern Europe6.11-1.14-1.43-0.350.09
CIS and Mongolia6.225.24-6.611.831.72
Developing Asia6.78-0.60-0.241.821.70
Middle East5.562.561.822.502.58
Western Hemisphere-1.81-8.57-9.22-7.58-4.87
Advanced economies2.60-3.48-2.550.120.88
2003 recovery results
Africa5.2823.290.50-1.38-1.65
Central and Eastern Europe12.845.055.356.185.92
CIS and Mongolia10.173.302.894.232.84
Developing Asia10.904.034.956.946.10
Middle East6.593.853.28-2.163.29
Western Hemisphere3.32-2.72-1.993.343.12
Advanced economies3.29-2.70-2.43-0.970.17
Import Volume
2001 recession results
Africa6.362.211.56-3.6922.10
Central and Eastern Europe1.82-6.06-5.33-5.67-2.53
CIS and Mongolia13.139.199.019.182.56
Developing Asia-0.17-6.57-8.71-7.52-5.79
Middle East7.201.895.364.445.44
Western Hemisphere2.10-3.47-3.27-4.73-3.51
Advanced economies-1.28-7.78-8.77-7.23-3.49
2002 recovery results
Africa4.23-0.3723.3620.95-6.15
Central and Eastern Europe8.622.091.631.712.30
CIS and Mongolia6.634.48-0.71-3.66-4.17
Developing Asia5.86-0.91-0.26-0.170.65
Middle East11.898.397.915.784.62
Western Hemisphere-7.20-12.10-12.47-9.43-4.45
Advanced economies2.17-4.08-2.63-0.580.12
2003 recovery results
Africa4.7325.460.17-2.52-4.06
Central and Eastern Europe13.196.196.337.807.30
CIS and Mongolia15.039.268.579.419.56
Developing Asia10.351.903.414.334.06
Middle East2.08-2.28-2.75-1.771.97
Western Hemisphere4.88-0.12-0.264.082.56
Advanced economies4.50-1.53-1.24-0.120.85
Inflation
2001 recession results
Africa18.4212.589.317.412.36
Central and Eastern Europe10.494.884.230.760.91
CIS and Mongolia16.43-8.29-1.56-2.83-2.03
Developing Asia4.911.080.430.010.10
Middle East1.39-2.81-3.20-2.24-1.85
Western Hemisphere7.011.52-2.370.31-1.01
Advanced economies2.530.430.100.11-0.05
2002 recovery results
Africa11.005.293.191.320.22
Central and Eastern Europe7.401.090.24-0.73-0.72
CIS and Mongolia11.141.180.830.870.19
Developing Asia7.172.452.342.132.22
Middle East2.34-2.97-3.05-2.32-2.72
Western Hemisphere6.731.671.690.660.26
Advanced economies2.170.030.050.220.01
2003 recovery results
Africa18.1812.340.271.030.21
Central and Eastern Europe5.12-1.35-1.61-0.97-0.49
CIS and Mongolia9.080.810.62-0.04-0.81
Developing Asia6.681.761.640.41-1.10
Middle East3.32-0.20-0.59-0.43-0.84
Western Hemisphere9.753.380.680.61-0.12
Advanced economies1.81-0.02-0.06-0.26-0.05
Current Account (in Percent of GDP)
2001 recession results
Africa-7.79-0.17-0.290.210.06
Central and Eastern Europe-5.01-0.250.04-0.050.67
CIS and Mongolia-2.895.514.300.75-0.81
Developing Asia0.422.352.442.141.86
Middle East6.836.682.400.56-0.84
Western Hemisphere-7.67-3.72-2.52-1.73-0.05
Advanced economies1.670.380.330.890.50
2002 recovery results
Africa-6.800.080.960.731.32
Central and Eastern Europe-5.55-0.87-0.16-0.33-0.19
CIS and Mongolia-2.680.760.102.171.38
Developing Asia-0.112.582.252.962.79
Middle East3.231.57-0.791.680.40
Western Hemisphere-6.62-1.021.001.840.73
Advanced economies2.421.711.210.930.76
2003 recovery results
Africa-5.022.382.362.641.89
Central and Eastern Europe-5.78-0.81-0.78-0.50-0.24
CIS and Mongolia-2.842.662.121.991.34
Developing Asia1.093.893.452.351.91
Middle East6.905.783.23-1.050.88
Western Hemisphere-4.601.930.790.92-0.04
Advanced economies2.450.840.660.25-0.07
Source: Author’s calculations.
Source: Author’s calculations.

Much lower levels of significance are obtained for the next-year forecasts where, in the case of the Advanced Economies, close to 60 percent of the β−estimates are positive and only 10 percent of the coefficients exceed two. Furthermore, the average R2−value now declines to a level near 0.10.

The R2 values do not in themselves quantify the degree of improvement in the WEO forecast from the Board to the published version. A better measure of this is the ratio of MSE-values based on t=1T(et,tApr)2/t=1T(et,tBoard)2 and t=1T(et+1,tSep)2/t=1T(et+1,tBoard)2, respectively, where T is the sample size. The final column in Table 10 show these ratios. Values below unity indicate that the WEO forecast gets more precise from the Board to the published version and the extent to which the ratio is below unity is a measure of the improvement. For the February/April same-year forecasts the average ratio is around 0.80 for the Advanced Economies. This declines to a value near 0.70 for the July/September same-year forecasts, but is closer to 0.95 for the next-year forecasts. These values suggest that much valuable information is learned about current-year economic growth between the time when the Board forecast is reported and the time of the official publication. Far less information is learnt about next-year economic growth between these dates as witnessed by the R2−values near 0.95.

Turning to the countries outside the Advanced Economies region, in general the percentage of positive coefficients in the Board-to-Publication revision regressions in (10) is somewhat lower as is the fraction of estimates that is statistically significant. In fact, only about 10–15 percent and 5–10 percent of the current-year and next year coefficient estimates generate positive t-values that exceed two. Interestingly, the MSE ratio tends to be somewhat lower than was found for the Advanced Economies, especially for the next-year forecasts, suggesting a significant improvement in the next year forecasts between the Board and the published forecasts for the other regions.

IX. Recent Performance of WEO Forecasts

Table 11 presents results for the recent performance of the WEO forecasts during the 2001 recession and 2002, 2003 recovery years. The recession and recovery year labels should be loosely interpreted as they are not uniformly accurate descriptors across the various regions. For example, although U.S. GDP growth was only 0.25 percent in 2001, this year saw average growth close to 8 percent in CIS and Mongolia, 4.3 percent in Africa and 3.5 percent in the Middle East—all examples of robust growth.

Table 11.Forecasts for G-7 Economies: Descriptive Statistics for Long Sample
VariableMeanForecast Errors
MeanStandard

Deviation
Absolute

Forecast Error
Forecast

Error
Standard

Deviation
Serial

Correlation1/
Real GDP
April current-year forecast analysis
Canada2.862.240.92-0.181.21-0.19
France2.341.790.64-0.131.000.11
Germany1.971.920.90-0.291.220.17
Italy1.941.990.91-0.211.20-0.07
Japan3.322.991.21-0.061.53-0.05
United Kingdom1.761.970.79-0.111.030.03
United States2.742.270.760.050.90-0.10
September next-year forecast analysis
Canada2.672.181.55-0.571.88-0.07
France2.181.551.07-0.551.50-0.09
Germany1.911.951.51-0.821.92-0.10
Italy1.922.021.44-0.511.870.13
Japan3.002.801.67-0.802.570.13
United Kingdom1.901.971.20-0.341.580.41
United States2.652.291.320.031.80-0.01
Inflation
April current-year forecast analysis
Canada4.673.671.050.121.330.28
France5.253.950.690.160.980.03
Germany3.292.110.59-0.080.75-0.07
Italy8.955.950.900.511.450.15
Japan2.504.471.04-0.451.750.16
United Kingdom7.355.821.170.291.64-0.41
United States4.202.600.38-0.050.550.10
September next-year forecast analysis
Canada4.703.881.490.182.020.41
France5.414.461.000.291.390.52
Germany3.091.950.64-0.120.780.26
Italy9.026.041.770.772.450.37
Japan2.354.551.70-0.502.81-0.05
United Kingdom7.235.941.420.692.230.38
United States4.302.710.84-0.111.220.38
Current Account
April current-year forecast analysis
Canada-4.6410.524.050.495.650.07
France7.1315.584.430.126.440.37
Germany6.2525.419.570.0013.720.28
Italy1.9714.836.64-1.119.18-0.14
Japan61.1449.1212.630.2616.54-0.05
United Kingdom-8.8513.915.130.827.350.03
United States-135.21154.9515.70-1.2818.97-0.15
September next-year forecast analysis
Canada-4.1611.334.661.086.870.52
France6.8114.746.76-0.488.770.51
Germany8.4224.7013.22-0.1719.230.42
Italy1.0415.0810.47-2.1614.340.33
Japan60.6649.6817.66-0.0922.690.29
United Kingdom-6.5613.626.272.238.670.37
United States-129.96153.1426.41-5.1337.840.15
Import Volume
April current-year forecast analysis
Canada6.277.354.851.695.95-0.05
France4.705.963.29-0.274.36-0.18
Germany4.955.633.75-0.875.33-0.09
Italy4.346.833.97-0.604.99-0.12
Japan5.358.043.64-0.424.39-0.18
United Kingdom4.985.062.970.453.53-0.07
United States7.037.833.961.374.96-0.07
September next-year forecast analysis
Canada5.856.905.451.706.620.12
France4.935.533.99-0.185.10-0.09
Germany5.065.083.78-0.234.560.08
Italy4.126.944.65-1.206.02-0.05
Japan4.867.795.25-0.746.590.09
United Kingdom4.835.073.510.244.200.12
United States6.997.874.881.615.80-0.06
Export Volume
April current-year forecast analysis
Canada5.466.193.381.054.700.05
France4.964.773.11-0.144.02-0.47
Germany5.005.793.54-0.444.76-0.18
Italy4.594.823.75-0.694.46-0.12
Japan5.206.544.190.855.16-0.22
United Kingdom3.973.842.79-0.173.54-0.44
United States5.757.592.480.243.45-0.38
September next-year forecast analysis
Canada6.125.654.221.585.440.18
France4.804.553.25-0.624.29-0.21
Germany5.185.373.680.054.82-0.21
Italy4.494.764.62-0.905.280.13
Japan5.326.655.110.806.07-0.29
United Kingdom4.133.702.97-0.293.460.00
United States5.837.703.510.544.670.01
Source: Author’s calculations.

First-order serial correlation coefficient.

Source: Author’s calculations.

First-order serial correlation coefficient.

Column one in Table 11 shows the mean value of the actual values computed across countries in the various regions. Columns two through five show the 21 average forecast error computed in April and September of the previous year (labeled (-1)) and April and September of the current year (labeled (0)).

First consider the GDP forecasts. The WEO forecasts overpredicted growth in 2001 in all regions with the exception of CIS and Mongolia and the September 2000 forecast of 2001 growth in the Middle East. There are clear patterns in the forecast revisions: predictions of 2001 GDP growth were not revised downwards to any noticeable degree between April and September 2000. However, there was a substantial reduction in the mean growth forecast between September 2000 and April 2001 and again from April to September 2001. For example, the mean forecast error of the September 2000 forecast of 2001 growth in the Advanced Economies was 2.17 percent. This gets reduced to 1.32 percent and 0.35 percent in the April and September 2001 forecasts, respectively.

Results for the recovery in 2002 bear some resemblance to the 2001 recession. The April and September previous-year (2001) WEO forecasts overpredicted growth in six of the seven regions, followed by significant adjustments in the April 2002 current-year forecast which gets the forecast right to within 0.4 percent of the actual value in four regions. One clear difference to the results for 2001 is that there is now evidence of small underpredictions in the September current year forecast for five of the seven regions, the only exceptions being Africa and Developing Asia for which the WEO forecasts on average overpredicted GDP growth.

The 2003 forecasts were very precise for Africa, central, and eastern Europe and developing Asia where all forecast errors lie within 0.7 percent of the realization. Once again the familiar pattern of previous-year April and September overpredictions followed by significant reductions that bring the prediction close to the actual value in the April and September same-year value is observed for the Advanced Economies.

Turning to the inflation forecasts, forecast errors are, unsurprisingly, more volatile in many of the regions than the corresponding GDP forecast errors. Nevertheless, some patterns emerge from the data. For Africa the previous year April forecast underpredicted the actual value in all three years by a wide margin ranging from 5.3 percent in 2002 to 12.6 percent in 2001. However, this forecast error comes down significantly between April and September of the previous year and gets further reduced in the subsequent same-year forecasts. Inflation forecasts were generally quite accurate for 2003. Moreover, the WEO inflation forecasts for the Advanced Economies are very accurate in all years. They fall within 0.45 percent of the actual value and are typically much closer than this to the outcome.

Finally, the WEO forecasts of the current account (as a percentage of GDP) show a general tendency to underpredict during the 2001–2003 period for countries in CIS and Mongolia, Developing Asia, Middle East, and the Advanced Economies. In general the numbers are not that large and the absolute value of the forecast error tends to get reduced as the point of the prediction draws closer to the time of the outcome.

X. Long-Run Forecasting Performance for G-7 Economies

Artis (1997) undertook a postmortem analysis of the WEO forecasting performance for the G-7 economies using data from the early seventies up to 1994. The decade that has passed since this analysis has produced new data that facilitates an analysis of the WEO forecasting performance around subsequent events.

As in the earlier analysis, we have data on five variables, namely GDP growth, inflation, export and import volume and the current account balance.9 In the case of the GDP growth and inflation data the current year forecasts date back to 1971 while next-year forecasts date back to 1973. In the case of the import and export volume and the current account balance, the current year forecasts only date back to 1972 while next-year forecast begin in 1973. Data on outcomes extend up to 2003 so we have between 33 and 31 annual observations. Not a very long sample by any common yardstick, but much longer than the WEO data and Artis’ original data.10

The WEO reports current year and next year forecasts in both April and September each year so a decision has to be made on which of these to use in the analysis. We follow Artis (1997) and focus on the current-year forecast published in April and the next-year forecast published in September.

A. GDP Growth

First consider the full-sample results for the GDP growth forecasts reported in Table 12. Current year mean absolute forecast errors range from 0.64 for France to 1.21 for Japan while the RMSFE-values vary from 0.90 (USA) to 1.53 (Japan). Forecast errors for all countries except the US have a slightly negative mean, representing over-predictions.

Table 12.G-7 Countries: Forecasting Performance Relative to Simple Benchmarks(Ratio of WEO RMSFE over the RMSFE-values produced by naïve random walk and recursive mean forecasts)
Random WalkRecursive Mean
Full SampleSubperiod 1Subperiod 2Full SampleSubperiod 1Subperiod 2
Real GDP
April current-year forecasts
Canada0.440.410.540.520.500.58
France0.470.440.570.520.500.62
Germany0.480.450.560.600.570.67
Italy0.410.360.660.520.480.70
Japan0.470.400.650.440.460.42
United Kingdom0.480.460.540.490.490.49
United States0.310.280.450.370.330.50
September next-year forecasts
Canada0.680.610.860.810.750.93
France0.780.671.030.870.741.17
Germany0.760.670.970.880.771.17
Italy0.690.641.010.820.771.06
Japan0.870.830.990.800.880.66
United Kingdom0.780.730.910.720.640.92
United States0.610.560.790.690.620.96
Inflation
April current-year forecasts
Canada0.670.630.790.350.450.26
France0.740.711.020.240.440.10
Germany0.540.490.650.330.390.27
Italy0.590.590.570.240.370.10
Japan0.540.530.930.370.430.16
United Kingdom0.360.340.740.250.300.13
United States0.360.350.480.200.270.11
September next-year forecasts
Canada0.930.930.950.490.660.31
France0.850.811.360.290.470.13
Germany0.610.460.840.390.350.41
Italy1.071.081.000.390.640.16
Japan0.850.831.280.520.590.25
United Kingdom0.520.501.070.350.420.17
United States0.780.750.980.410.530.23
Current Account
April current-year forecasts
Canada0.960.990.960.530.880.51
France0.940.641.050.410.710.39
Germany0.830.650.850.530.320.59
Italy0.790.500.820.610.560.61
Japan0.780.680.800.330.290.35
United Kingdom0.780.760.800.520.670.47
United States0.440.610.410.120.220.11
September next-year forecasts
Canada1.130.921.150.600.750.60
France1.200.991.260.591.120.55
Germany1.141.091.150.770.580.81
Italy1.370.981.430.941.070.93
Japan1.080.851.130.450.340.48
United Kingdom0.940.960.930.630.750.58
United States0.830.810.830.250.280.24
Import Volume
April current-year forecasts
Canada0.630.530.860.760.641.02
France0.510.250.770.670.331.01
Germany0.770.540.860.910.611.04
Italy0.510.440.610.660.540.92
Japan0.420.350.610.490.430.66
United Kingdom0.570.440.850.650.520.88
United States0.470.360.790.570.440.91
September next-year forecasts
Canada0.730.631.000.910.791.25
France0.640.540.810.840.721.05
Germany0.720.680.740.840.770.89
Italy0.600.560.670.780.681.02
Japan0.660.590.850.720.621.06
United Kingdom0.680.660.740.740.710.87
United States0.550.450.940.660.551.05
Export Volume
April current-year forecasts
Canada0.610.520.800.710.620.91
France0.610.450.730.760.501.03
Germany0.570.350.830.750.451.08
Italy0.670.620.730.840.780.92
Japan0.540.480.630.750.640.97
United Kingdom0.650.450.910.790.531.14
United States0.430.370.530.430.360.53
September next-year forecasts
Canada0.820.691.060.900.821.02
France0.710.630.800.890.781.01
Germany0.630.550.900.770.680.98
Italy0.820.730.931.040.981.10
Japan0.610.620.610.850.820.92
United Kingdom0.730.640.880.840.721.09
United States0.560.440.740.520.370.96
1/ Source: Author’s calculations.
1/ Source: Author’s calculations.

Biases in the next-year forecast error are again negative for six of the seven economies. Next year biases are also greater than the biases observed for the current-year forecasts. There is little evidence of serial correlation in the full sample forecast errors, the only exception being the next-year forecasts for the United Kingdom which generate a first-order serial correlation coefficient of 0.41.

We measure forecasting performance relative to two naive benchmarks by computing the ratio of RMSFE-values of the WEO forecasts relative to a random walk forecast that uses the most recent (previous year) observation and relative to the recursive sample average of the target variable computed using the most recently available data (again the outcome during the previous year). The results, reported in Table 13, show that the WEO forecasts generate RMSFE-values around 40 percent the size of the random walk forecast or around 50 percent of the size of the RMSFE-value associated with the recursive mean forecast. Compared to the current year ratios, the next year RMSFE ratios increase by between 0.10 and 0.20. This means that the next year RMSFE ratios are close to 0.60 relative to the random walk forecasts while they fall between 0.60 and 0.80 for the recursive mean forecasts.

Table 13.G-7 Countries: Predictive Efficiency Test Results(t-statistics)
April Current-Year ForecastSeptember Next-year Forecast
Bias 1/Serial

Correlation 2/
Lagged

Realization 3/
Bias 1/Serial

Correlation 2/
Lagged

Realization 3/
Real GDP
Canada-0.84-1.38-1.12-1.76-0.32-0.15
France-0.761.03-1.22-2.20-0.42-1.75
Germany-1.410.99-0.40-2.62-0.45-0.86
Italy-1.02-0.41-0.89-1.590.46-0.28
Japan-0.24-0.23-1.30-1.821.49-0.77
United Kingdom-0.630.141.41-1.221.930.60
United States0.30-0.45-2.240.09-0.07-1.45
Post-1990 Sub Sample
Canada-0.79-0.63-0.02-1.540.70-0.03
France-1.24-0.621.53-2.25-1.55-1.83
Germany-0.530.111.36-2.34-1.83-2.23
Italy-2.04-0.81-0.95-4.40-0.92-0.80
Japan0.03-0.30-0.13-2.01-0.83-1.29
United Kingdom-0.120.85-0.03-1.660.660.08
United States0.931.541.190.330.980.60
Inflation
Canada0.511.360.220.501.591.84
France0.950.180.811.192.661.27
Germany-0.59-0.360.48-0.841.290.53
Italy2.150.38-1.261.852.15-0.01
Japan-1.521.38-0.86-1.01-0.11-0.32
United Kingdom1.04-2.261.051.821.400.96
United States-0.500.38-1.25-0.521.151.33
Post-1990 Sub Sample
Canada-1.01-1.170.83-2.300.681.23
France-1.020.762.10-2.300.721.88
Germany-0.490.660.61-0.550.190.03
Italy2.502.212.482.362.783.11
Japan-3.52-0.890.17-3.20-0.56-0.03
United Kingdom-0.02-2.13-1.31-0.030.611.62
United States-1.110.472.24-4.820.881.27
Current Account
Canada0.480.341.020.883.192.30
France0.112.34-0.66-0.312.94-0.56
Germany0.001.00-0.62-0.053.560.15
Italy-0.68-0.73-1.83-0.851.80-0.50
Japan0.09-0.27-2.33-0.021.48-1.46
United Kingdom0.630.20-0.541.483.41-0.17
United States-0.38-0.80-0.09-0.760.600.95
Post-1990 Sub Sample
Canada0.54-2.17-1.041.15-1.14-1.95
France0.55-0.22-1.830.091.08-0.50
Germany-0.483.871.18-0.772.571.60
Italy-1.121.63-0.56-0.650.79-1.13
Japan-0.39-1.660.32-0.32-0.33-0.49
United Kingdom1.41-2.510.023.400.151.95
United States-0.080.802.20-0.401.722.29
Import Volume
Canada1.68-0.24-0.751.480.67-0.18
France-0.35-1.06-2.05-0.20-0.42-1.50
Germany-0.93-0.54-0.57-0.280.35-0.24
Italy-0.68-0.55-1.63-1.13-0.22-1.44
Japan-0.54-1.01-1.44-0.630.39-1.21
United Kingdom0.72-0.420.680.320.570.75
United States1.63-0.43-0.161.60-0.32-0.05
Post-1990 Sub Sample
Canada1.080.170.620.98-1.10-0.47
France-1.20-0.830.18-1.05-0.86-0.64
Germany-0.83-0.23-0.26-0.090.480.81
Italy-0.181.341.21-1.46-0.88-1.18
Japan-0.162.641.82-0.190.25-1.36
United Kingdom-0.13-0.860.20-0.02-1.38-0.18
United States1.063.012.111.61-0.58-0.91
Export Volume
Canada1.290.31-0.961.691.020.51
France-0.19-3.01-2.17-0.81-1.57-1.48
Germany-0.52-1.16-1.250.06-1.11-1.30
Italy-0.89-0.65-2.23-0.960.77-0.28
Japan0.94-1.39-3.270.74-1.79-3.35
United Kingdom-0.28-2.71-0.84-0.47-0.01-1.33
United States0.39-2.97-1.420.640.03-0.20
Post-1990 Sub Sample
Canada0.781.532.240.810.670.78
France-0.36-0.622.03-0.76-0.42-1.09
Germany-1.211.072.30-0.350.90-0.23
Italy-0.881.010.58-1.30-1.40-1.38
Japan0.122.092.94-0.220.830.00
United Kingdom-0.570.041.65-1.14-3.29-2.40
United States-0.162.741.35-0.09-5.37-5.73
Sources: Author’s calculations.

t-statistics for the constant α^ in equation (1).

t-statistics for the coefficient β^ in equation (2).

t-statistics for the lagged realization in a regression of the forecast error on the lagged realization.

Sources: Author’s calculations.

t-statistics for the constant α^ in equation (1).

t-statistics for the coefficient β^ in equation (2).

t-statistics for the lagged realization in a regression of the forecast error on the lagged realization.

To see if the (relative) performance of the WEO forecasts has changed over time, we divided the sample into two equally long subintervals. The results suggest that the RMSFE-performance of the current-year and next-year WEO forecasts (relative to the naive forecasts) has worsened in most cases, sometimes by quite a significant amount. For example, in the second subsample the recursive mean now produces lower RMSFE values than the WEO next-year forecasts for France, Germany, and Italy. This could be related to the lower global volatility of GDP growth which tends to favor more smooth forecasts such as the recursive mean estimate. As is evident from the table, it has become more difficult to beat the simple benchmark forecasts during the subsequent decade.

Efficiency tests based on univariate regressions of forecasts errors on (1) a constant; (2) a constant and the lagged error; (3) a constant and the lagged realization are reported in Table 14. The first three columns present the current-year t-statistics for the constant, the lagged error and the lagged realization, while columns four through six show results for the next-year forecasts. Standard errors are heteroskedasticity and autocorrelation consistent using a Bartlett kernel with up to 2 lags, c.f. Newey and West (1987). The most notable finding is that the positive bias found in next-year forecast errors is significant for two of the countries, namely France and Germany.11

Table 14.Consensus Forecasts: Data Coverage(Number of observations)
Real GDPInflationCurrent Account
Current-

Year
Next-

Year
Current-

Year
Next-

Year
Current-

Year
Next-

Year
G-7 Countries (March/September)
Canada141314131413
France141314131413
Germany141314131413
Italy141314131413
Japan141314131413
United Kingdom141314131413
United States141314131413
Latin America (February/August)
Argentina111011101110
Brazil141314131413
Chile111011101110
Colombia111011101110
Mexico141314131413
Peru111011101110
Venezuela111011101110
Asia (March/September)
China989898
Hong Kong SAR131213121212
India989898
Indonesia131213121312
Korea141314131413
Malaysia131213121312
Singapore131213121312
Taiwan Province of China141314131413
Thailand131213121312
Source: Author’s calculations.
Source: Author’s calculations.

B. Inflation Forecasts

Inflation is generally viewed as one of the most difficult macroeconomic time series to predict, perhaps due to the evidence of instability, related to the effects of changes in monetary policy. However, for most G-7 countries the RMSFE values for inflation shown in Table 12 are quite similar to the values for GDP growth. As expected the mean absolute forecast errors are higher for the next ear than for the current-year forecasts. Furthermore, there is now evidence of positive serial correlation at the next-year horizon for six of seven economies.

Theil statistics for the ratio of RMSFE-values of the current-year WEO forecasts relative to the two naive forecasts lie between 0.36 and 0.74 (relative to the random walk forecasts) and are even lower relative to the recursive mean forecast against which they range between 0.20 and 0.37, with the lowest values observed for the United States, c.f. Table 13. Turning to the next-year forecasts, all Theil statistics remain well below unity. The relative performance of the WEO next-year forecasts is only marginally worse than the current year performance. For this variable there is systematic evidence that the WEO forecasts perform worse over time relative to the random walk forecasts, but improve relative to the recursive mean forecasts. This could well be related to structural breaks in the underlying inflation series.

Efficiency and Mincer-Zarnowitz regressions (the former shown in Table 14) only indicate scant evidence of biases in the current-year inflation forecasts, an exception being Italy.

However there is some evidence of serial correlation in the forecast errors for France, Italy, and the United Kingdom.

C. Import and Export Volume, Current Account

For the current account balance forecasts, the best naive forecast clearly comes from the random walk model and the performance against this benchmark deteriorates somewhat in the second half of our sample (and exceeds unity in one case).

Theil statistics for the WEO current- and next-year forecasts of import and export volume compared to random walk and recursive mean forecasts were all less than unity c.f. Table 13. Furthermore, the subsample analysis again shows that it has become increasingly difficult over time to beat the naive benchmarks. For both trade volume variables there are three or four instances where the RMSFE ratio exceeds unity when measured against the recursive mean forecasts. This suggests that a simple smooth forecast would perform well for around half of the countries. When measured against the random walk forecast, the RMSFE ratio is always below unity and the WEO forecasts dominate.

D. Performance After 1990

A separate analysis of the WEO forecasting performance after 1990 is warranted due to the ongoing debate on whether the productivity of the US economy has undergone a structural break and whether similar permanent or highly persistent structural changes could have affected economies such as Germany (due to the reunification in 1989) and Japan. To explore this issue we computed separate results for the post-1990 period. The outcome is reported in Table 14. There is evidence from the next-year forecasts that the WEO systematically overpredicted economic growth after 1990 for all the European G-7 economies, i.e. France, Germany, Italy, and (to a smaller degree) the United Kingdom. Japanese growth was also overpredicted after 1990 and the associated t-values are very high for most economies despite the short sample.

In contrast, US growth was underpredicted after 1990 although there is no corroborating statistically significant evidence. The same is true for the tests for serial correlation which generally fail to have sufficient power—this is unsurprising in light of the short sample used in this subsample analysis.

A similar analysis of inflation rates after 1990 shows some very interesting patterns in the next-year forecast errors. Although next-year inflation forecasts do not appear to be biased over the full sample from 1973–2003, the story is very different for the post-1990 sample. Inflation was strongly and significantly overpredicted for Canada, France, Japan, and the United States and was only underpredicted (again at a significant margin) for Italy. Furthermore, there is evidence of serial correlation in the forecast errors for Italy.

These findings have at least two possible explanations. One possibility is that the output growth and inflation series have been subject to structural breaks. This is a real possibility given the systematic evidence of structural instability underlying many reduced form models used for forecasting, c.f. Stock and Watson (1996). Another possibility is that the conditioning information underlying some of these models—such as an assumption that the output gap will be eliminated over a reasonably short span of time—gives rise to biases.

XI. Comparison of WEO and Consensus Forecasts

A comparison of forecasts to subsequent outcomes—as we have proceeded with so far—is an important exercise or “reality check” that allows us to test if basic efficiency properties are satisfied by the forecasts. This exercise clearly has its own limitations, however. For example, it is not evident what constitutes a good forecast in absolute terms. Some series may be intrinsically very difficult to predict (inflation comes to mind) because they are affected by large exogenous shocks and/or shifts in economic policy whose effect are difficult to predict in advance. Conversely, a forecast can be very uninformative but lead to errors that do not appear to violate efficiency properties such as unbiasedness and absence of serial correlation.

To address this issue, it is therefore highly informative to compare the WEO forecasts to alternative forecasts such as those produced by a highly reputed source such as the Consensus forecasts. Forecasters included in the Consensus survey faced similar difficulties as the WEO forecasters—e.g., the higher than expected productivity growth for the United States economy or the absence of large, global inflationary shocks during most of the 1990s—and therefore serve as a yardstick against which the WEO forecasts can be measured.

A second advantage of the Consensus data is that it allows us to address whether a forecaster could have done better by using both the WEO and Consensus forecasts. This is highly relevant for followers of both sets of forecasts and is ultimately also relevant for the WEO exercise—if there exists a simple combination of the WEO and Consensus forecasts that improves upon the WEO forecasts, the WEO forecasts should either be based on this combination or the potential source of inefficiency in the WEO forecasts should be identified with a view toward improving them.

This discussion motivates a comparison of the WEO forecasts with the Consensus forecasts. This is clearly a useful exercise: The Consensus forecasts constitute readily available public information so the WEO forecasts must be justified against this alternative source of forecasting information.

A. Consensus Data

To investigate the relative performance of the WEO and Consensus forecasts, we obtained Consensus forecast data on GDP growth, inflation and the current account balance over the period 1990–2003. The data covers all the G-7 economies, seven Latin American economies (Argentina, Brazil, Chile, Colombia, Mexico, Peru, and Venezuela) and nine Asian economies (China, Hong Kong, India, Indonesia, Malaysia, Singapore, South Korea, Taiwan Province of China, and Thailand).

In the baseline scenario, Consensus forecasts are measured in March (in the case of the current-year forecasts) and September (in the case of next year forecasts) except for the Latin American economies where data coverage is limited for these months. For this reason the February and August Consensus forecasts were used for these economies. The Consensus forecast is computed as the mean forecast across participants in a given monthly survey.

We shall refer to the March current-year Consensus forecast as y^t,tcons while the September next-year Consensus forecast is denoted y^t+1,tcons. Although Consensus forecasts are now available on a monthly basis, the March and September Consensus forecasts are the forecasts that are based on information whose timing is most similar to the WEO April current-year (denoted y^t,tWEO) and September next-year (y^t+1,tWEO) forecasts so this comparison was deemed most appropriate in relation to measuring the information content of the two sets of forecasts. For completeness we shall later report the outcome of a sensitivity analysis that changes the timing of the Consensus forecasts.

Table 15 shows the data coverage. The data is complete for the G-7 economies where 14 current-year forecasts and 13 next-year forecasts (1991–2003) are available. The data is somewhat less complete for Latin America where only Brazil and Mexico have full coverage while countries such as Argentina, Chile, Colombia, Peru, and Venezuela only have 10 or 11 forecasts. In the Asia region the data coverage is excellent for most of the countries except for China and India where only 8 or 9 forecasts are available.

Table 15.Correlations Between WEO and Consensus Forecasts
Forecast ErrorsForecast Levels
Current-YearNext-YearCurrent-YearNext-Year
GDP Growth
G-7 countries
Canada0.9760.9820.9850.931
France0.9500.9880.9760.939
Germany0.9460.9700.9670.838
Italy0.9670.9850.9640.893
Japan0.9650.9680.9710.908
United Kingdom0.9050.9670.9770.833
United States0.9500.9670.9780.745
Latin America
Argentina0.9120.9950.9500.901
Brazil0.9560.9160.9270.662
Chile0.6290.9710.6620.842
Colombia0.9790.9710.9390.831
Mexico0.9850.9820.9620.465
Peru0.8450.9850.4640.719
Venezuela0.9680.9770.9830.850
Asia
China0.8330.8670.9260.920
Hong Kong SAR0.9130.9810.8250.954
India0.9840.9590.9030.694
Indonesia0.9860.9850.9920.952
Korea0.9930.9960.9900.989
Malaysia0.9860.9920.9750.975
Singapore0.9760.9960.9500.976
Taiwan Province of China0.0970.9800.9550.919
Thailand0.9920.9940.9940.971
Inflation
G-7 countries
Canada0.8280.9260.9810.961
France0.8230.9600.9660.980
Germany0.8870.9040.9870.946
Italy0.9090.9400.9900.976
Japan0.7900.9240.9800.970
United Kingdom0.8920.8810.9780.976
United States0.9470.9320.9920.960
Latin America
Argentina0.3150.9910.9620.995
Brazil0.4590.9570.9080.992
Chile-0.0440.7300.9670.976
Colombia0.8590.7400.9860.976
Mexico0.6740.9660.9370.952
Peru-0.927-0.1620.8860.980
Venezuela0.8990.9350.9420.939
Asia
China0.8270.9820.9980.985
Hong Kong SAR0.8870.4960.9860.829
India0.9210.9460.8740.913
Indonesia0.9390.9960.9870.984
Korea0.9090.9210.9680.892
Malaysia0.4680.7330.7690.723
Singapore0.7560.7240.8780.406
Taiwan Province of China0.9220.6550.9590.819
Thailand0.9100.8120.9820.819
Current Account Balance
G-7 countries
Canada0.9400.9470.9710.968
France0.9570.8730.9900.935
Germany0.9590.9440.9740.892
Italy0.8840.9490.9560.954
Japan0.9280.9680.9640.971
United Kingdom0.9240.9330.9380.928
United States0.7970.8830.9920.991
Latin America
Argentina0.8790.9220.9750.957
Brazil0.9280.9840.9750.988
Chile0.9740.9840.9700.944
Colombia0.9000.9680.8660.901
Mexico0.8530.9650.9250.941
Peru0.9270.9500.9510.924
Venezuela0.9220.9420.9090.890
Asia
China0.9520.9430.9400.927
Hong Kong SAR0.8860.6870.9000.613
India0.9240.8920.9100.837
Indonesia0.9390.9370.9900.980
Korea0.9880.9900.9910.985
Malaysia0.9380.9640.9680.965
Singapore0.6690.8590.9410.952
Taiwan Province of China0.7550.9020.8600.633
Thailand0.9480.9690.9900.985
Source: Author’s calculations.
Source: Author’s calculations.

As a first check of the quality of the data, we report in Table 16 pairwise correlations between current-year (etWEO=yty^t,tWEO and etcons=yty^t,tcons) and next-year (et+1,tWEO=yt+1y^t+1,tWEO and et+1,tcons=yt+1y^t+1,tcons) forecast errors as well as the correlation between the time-series of the forecasts levels ((y^t,tWEO,y^t,tcons) or (y^t+1,tWEO,y^t+1,tcons)). In all countries and for both current year and next year forecasts and forecast errors the WEO and Consensus values are strongly positively correlated. Interestingly, the weakest correlation (0.75) is found for United States next year forecast levels. Estimated correlations are generally highest for current-year forecasts (where they all exceed 0.96) compared to next year forecasts.

Table 16.Biases in WEO and Consensus Forecasts(Mean forecast errors)
Current-yearNext-year
WEOConsensusWEOConsensus
Real GDP
G-7 countries
Canada-0.214-0.173-0.580-0.520
France-0.202-0.271-0.800-0.685
Germany-0.127-0.045-1.117-0.856
Italy-0.429-0.469-1.020-0.956
Japan0.0120.060-1.1270.834
United Kingdom-0.023-0.132-0.476-0.448
United States0.1750.2160.2310.132
Latin America
Argentina-0.165-0.956-2.461-2.406
Brazil0.017-0.049-0.598-0.983
Chile-0.332-0.719-0.949-0.963
Colombia-0.996-0.909-1.354-1.685
Mexico-0.340-0.504-1.880-1.480
Peru0.9501.601-0.535-0.099
Venezuela-0.827-1.932-3.783-3.514
Asia
China0.6090.2110.680-0.228
Hong Kong SAR-0.503-0.056-0.548-0.185
India0.3030.4790.011-0.551
Indonesia-0.161-0.342-1.546-1.544
Korea0.6380.469-0.264-0.308
Malaysia-0.109-0.202-0.542-0.658
Singapore0.337-0.0240.336-0.305
Taiwan Province of China-0.353-0.526-0.611-0.702
Thailand-0.492-0.441-1.312-1.236
Inflation
G-7 countries
Canada0.002-0.088-0.202-0.340
France0.0430.013-0.051-0.284
Germany0.0910.0580.116-0.030
Italy0.0820.1050.2520.142
Japan-0.0700.015-0.242-0.256
United Kingdom-0.0730.075-0.256-0.365
United States0.0050.025-0.296-0.370
Latin America
Argentina-1.456-5.668-2.018-1.838
Brazil288.62172.307326.132216.726
Chile0.150-0.242-0.163-0.321
Colombia0.302-0.547-0.796-0.249
Mexico2.3341.6344.0683.680
Peru1.415-3.8170.094