Journal Issue
Share
Article

An Evaluation of the World Economic Outlook Forecasts

Author(s):
Allan Timmermann
Published Date:
March 2006
Share
  • ShareShare
Show Summary Details

I. Introduction and Summary

The World Economic Outlook (WEO) is a key source of forecasts of global economic activity and is a key vehicle in the IMF’s multilateral surveillance activities. It is published twice a year in April and September. Given the central role of the WEO forecasts, it is important that they are periodically evaluated to assess their usefulness, and to look for ways to improve the forecasting process. This report is the fourth in a series of such evaluations (following Artis, 1996; Barrionuevo, 1993; and Artis, 1988).

This report analyzes the forecast performance for five key variables—real GDP growth, inflation, the current account balance, and import and export volume growth—from 1990 to 2003, the last year for which actual data were available when the report was initiated. The report incorporates state-of-the-art techniques that shed light on the accuracy of WEO forecasts from new angles, and features three main novel aspects.

  • First, it analyzes forecasts for 178 countries in seven economic regions (Africa, Central and Eastern Europe, the Commonwealth of Independent States (CIS) countries, and Mongolia, developing Asia, the Middle East, the Western Hemisphere, and the advanced economies) since 1990. Earlier evaluations had focused on forecasts for the same variables for only the G-7 countries and regional aggregates.
  • Second, it includes an extensive comparison between the accuracy of WEO forecasts and Consensus forecasts. The latter is a widely used source that compiles the forecasts of economists working in the private sector. Through this comparison, the report assesses WEO forecasts not just against absolute benchmarks, but also against a relative benchmark of other forecasters.
  • Third, it considers the revisions to the forecasts, both over time and within each forecast round. The latter is important because there is a long gestation lag in the preparation of the forecasts in each round, and it is important to know the gains—in terms of accuracy—of frequent forecast updates.

This summary highlights the main findings of the report.

A. How Accurate Are WEO Forecasts?

The first part of the paper examines selected aspects of the WEO forecast performance. In all cases, the analysis considers current-year and next-year forecasts prepared in April and September for a variable. (For example, the April and September 2005 WEOs have projections for 2005 (current year) and 2006 (next year).) Overall, the report finds that WEO forecasts for many variables in many countries meet the basic forecasting quality standards in some, if not all, dimensions. The paper also finds some important issues, which are discussed on a variable-by-variable basis.

  • Real GDP growth. WEO forecasts for real GDP growth display a tendency for systematic overprediction—that is, predicted growth, on average, tends to exceed actual growth (Table S1). From a statistical perspective, these biases are most significant in the next-year forecasts. The results also indicate that systematic overpredictions of real GDP growth are particularly prevalent in forecasts for countries with an IMF program. This tendency for overprediction of growth performance is also persistent over time. The evidence suggests that WEO forecasts for some countries could be improved if more attention were paid to important international linkages. In particular, forecasts of U.S. GDP growth are positively and significantly correlated with current-year forecast errors of output growth in a substantial number of advanced economies. (The forecast of German GDP growth also has predictive power over output growth forecast errors in some regions.) The report also finds that, in some cases, accuracy problems appear related to the standing WEO assumption that the output gap is eliminated after five years. In particular, the paper notes a predominant negative relationship between the output gap and the forecast error in the GDP growth, notably for Germany, France, and Italy.
  • Inflation. The report finds a bias toward underprediction of inflation, with these biases significant in the next-year forecasts for many African, Central and Eastern European, and Western Hemisphere countries. The underprediction bias is generally found to be weaker in the current-year forecasts. With regard to their predictability, there is evidence that the next-year inflation forecast errors are often linked to U.S. GDP forecasts.
  • External current account balances. There appear to be fewer problems in the forecasts for current account balances as percentages of GDP, except for April next-year forecast errors, which, in some cases, are significantly biased or serially correlated. Moreover, general patterns in the direction of biases are not apparent.

Table 1.Overview of Forecast Accuracy(Averages across regions for April current-year forecasts and September next-year forecasts; fraction of economies in region where statistics are significant in parenthesis)
Real GDP GrowthCPI InflationCurrent Account Balance3/
Bias1/Serial

Correlation2/
Bias1/Serial

Correlation2/
Bias1/Serial

Correlation2/
Advanced Economies-0.04/-0.36

(0.07/0.21)
0.21/0.24

(0.03/0.14)
-0.08/-0.13

(0.00/0.07)
0.20/0.33

(10.07/0.17)
0.25/0.22

(0.07/0.11)
0.26/0.35

(0.07/0.29)
Africa-1.17/-1.48

(0.25/0.33)
0.21/0.03

(0.15/0.29)
0.60/1.00

(0.13/0.16)
0.29/0.26

(0.06/0.14)
-0.21/-0.67

(0.02/0.09)
0.27/0.24

(0.13/0.09)
Central and Eastern Europe-1.17/-1.40

(0.07/0.07)
0.37/0.34

(0.13/.20)
2.30/3.27

(0.13/0.16)
0.39/0.39

(0.33/0.33)
20.95/8.50

(0.00/0.00)
0.30/0.33

(0.07/0.13)
Commonwealth of Independent States-1.93/-2.39

(0.15/0.00)
0.31/0.46

(0.00/.15)
126.95/153.65

(0.00/0.20)
0.69/0.74

(0.69/0.77)
-7.49/1.51

(0.00/0.08)
0.24/0.39

(0.08/0.23)
Developing Asia-0.38/-0.53

(0.05/0.04)
0.25/0.31

(0.00/.13)
1.06/0.62

(0.00/0.08)
0.33/0.36

(0.15/0.21)
0.58/1.11

(0.11/0.09)
0.28/0.34

(0.05/0.09)
Middle East-1.66/-1.34

(0.08/0.07)
0.37/0.31

(0.15/.21)
-0.86/-1.15

(0.00/0.08)
0.28/0.27

(0.15/0.00)
-1.36/-1.69

(0.00/0.00)
0.23/0.25

(0.09/0.00)
Western Hemisphere-0.64/-1.16

(0.13/0.24)
0.23/0.21

(0.06/.21)
1.78/1.60

(0.13/0.27)
0.38/0.29

(0.22/0.15)
-0.61/-.95

(0.03/0.16)
0.25/0.27

(0.10/0.16)
Memorandum
G-7 Economies
Canada-0.21/0.58-0.14/0.260.0/-0.20-0.26/0.19-0.01/0.350.43/0.51
France-0.20/-0.80-0.38/-0.230.04/-0.050.32/0.630.14/0.080.38/0.64
Germany-0.13/-1.120.05/-0.260.09/0.120.21/0.37-0.08/-0.350.22/0.46
Italy-0.43/-1.02-0.20/-0.070.08/0.250.08/0.18-0.29/-0.25-0.25/0.38
Japan0.01/-1.03-0.01/0.23-0.07/-0.24-0.19/-0.03-0.02/-0.02-0.06/0.01
United Kingdom-0.02/-0.48-0.60/0.46-0.07/-0.26-0.09/-0.150.33/0.51-0.09/-0.04
United States0.17/0.230.16/0.230.01/-0.30-0.06/0.150.01/-0.12-0.59/-0.16
Source: Author’s calculations.

Mean forecast error.

Coefficient on lagged forecast error in a regression of the forecast error on a constant and its lagged realization.

In percent of GDP.

Source: Author’s calculations.

Mean forecast error.

Coefficient on lagged forecast error in a regression of the forecast error on a constant and its lagged realization.

In percent of GDP.

Besides the basic accuracy of WEO forecasts, the report also examined a number of other issues of interest.

Directional accuracy of forecasts. The results suggest that the WEO forecasts are quite successful in predicting the directional change for current-year real GDP growth and inflation, but somewhat less so for next-year forecasts.

  • Performance of WEO forecasts during the 2001 downturn. WEO forecasts of GDP generally overpredicted growth in 2001 in all regions, which is consistent with the broad patterns among forecasters in earlier downturns. For 2002, the April and September next-year WEO forecasts prepared in 2001 overpredicted growth in six of the seven regions, although revisions in the April 2002 WEO greatly reduced the forecast errors in four regions.

Revisions from board to published forecasts. WEO forecasts are published twice a year in April and September. Prior to publication, a first set of predictions is presented to the IMF Executive Board in February and July. Subsequently, the forecasts are revised before they are published.

These revisions add considerable informational value. For the February/April same-year forecasts, the average reduction in the forecast error is around one-fifth for the advanced economies. The reduction is nearly 30 percent for the July/September same-year forecasts, but only 5 percent for the next-year forecasts.

B. Long-Run Forecasting Performance for G-7 Countries

Taking advantage of the fact that a longer dataset starting in the early seventies exists for the G-7 economies, the report assesses WEO forecasts for these economies in more detail. Overall, the results suggest that the forecast accuracy has deteriorated somewhat since the last evaluation (Artis, 1996). In particular, WEO forecasts systematically and significantly overpredicted economic growth for all the European G-7 economies and Japan during 1991–2003. In contrast, U.S. growth was underpredicted after 1990, although the bias was not found to be statistically significant. In contrast, inflation was strongly and significantly overpredicted for Canada, France, Japan, and the United States during the 1990s and 2000s, although it was underpredicted by a significant margin for Italy.

These findings have at least two possible, not mutually exclusive, explanations. One is that output growth and inflation have been subject to structural breaks, such as a break toward higher productivity growth in the United States. Another possibility is that the underlying assumptions—such as the assumption that the output gap will be eliminated over a 5-year period—has led to biases.

C. Comparison of WEO and Consensus Forecasts

The report compared the WEO projections to Consensus forecasts for GDP growth, inflation, and the current account balance over the period 1990–2003.2 The data covers all the G-7 economies, seven Latin American economies (Argentina, Brazil, Chile, Colombia, Mexico, Peru, and Venezuela) and nine Asian economies (China, Hong Kong SAR, India, Indonesia, Korea, Malaysia, Singapore, Taiwan Province of China, and Thailand).

Overall, the comparison suggests that the forecast performance of the WEO is similar to that of the Consensus forecast—the current-year WEO forecasts of GDP growth in the G-7 economies are generally less biased than the current-year Consensus forecasts, but the bias in the next-year forecasts is stronger in the WEO than in the Consensus across the board. The paper highlights, however, that the timing of the comparison with the Consensus forecast matters. WEO current-year forecasts generally perform quite well against current-year Consensus forecasts reported in March and perform considerably better against the February Consensus forecasts. However, given the relatively long gestation lag in the preparation, they tend to perform considerably worse against the Consensus forecasts reported in April. With the possible exception of next-year inflation forecasts, there is little systematic evidence that the WEO forecasts could be improved by modifying them to account for information embodied in the Consensus forecasts.

D. Recommendations

The report makes the following recommendations to improve the WEO forecasting process:

  • Timeliness of information is key to forecasting performance. There are systematic gains in forecasting accuracy from using the latest available information. These gains were found in both the comparison of the forecasts of the Executive Board version to those of the published version of the WEO and in the comparison between the WEO and the Consensus forecasts. Clearly, the updating process is already adding significant value, especially for the G-7 economies, but more could be done. It is therefore important that IMF country economists update their projections just before publication.
  • Continuous monitoring of forecasting performance. The empirical analysis indicated structural instability in some of the underlying variables, especially real GDP growth and inflation, which is consistent with the broad evidence of instability in macroeconomic variables provided in academic studies. Given the presence of what appears to be systematic biases in forecasting performance for output growth and inflation, particularly after 1990, the possibility of instituting real-time forecasting performance indicators should be explored.
  • Use bias-adjusted forecasts as a guidance. The simplest and most obvious approach to improving forecasts is to shrink the forecast toward its bias-corrected value. While simple to implement if the bias can be precisely estimated, this approach may also be too mechanical and suffer from its own deficiencies, including, for example, the assumption that the bias remains constant through time. Nevertheless, a comparison of unadjusted forecasts with bias-adjusted forecasts can help in enhancing our understanding of the magnitude and direction of any biases that may exist.
  • Forecasts of risk. In view of the inherent uncertainty associated with forecasts, the report strongly recommends that, in the future, the WEO incorporate recent advances in modeling and forecasting risk/uncertainty, including, for example, by presenting the full probability distribution of key variables over time.
  • Review the output gap assumption. The WEO forecasts are based on scenarios that assume that the output gap is removed within a relatively short period. Since some of the countries with the largest output gaps were also found to be countries for which the WEO forecasts systematically overpredicted output growth, this could be a concern. Hence, an analysis that explores the cost and benefits of using this practice is called for. Also, more frequent reviews of estimates of potential output growth may be needed.

The report is organized as follows. Section II describes the principal dataset containing the WEO forecasts and outcomes. Section III introduces the statistical methods used to test the optimality properties of the WEO forecasts. This is followed by two empirical sections that go through the basic characteristics of the forecasts (Section IV) and statistical significance of tests of forecast inefficiencies (Section V). Section VI presents evidence on predictability of WEO forecast errors by means of a range of instruments, such as the WEO prediction of U.S. and German GDP growth, oil prices, the output gap, and the global current account discrepancy. Section VII conducts directional-accuracy tests, while Section VIII studies the information reflected in the process whereby forecasts are revised and updated from their discussion by the Executive Board to the published version of the WEO. Section IX considers the performance of the WEO forecasts during the most recent recession and recovery years; while, conversely, Section X looks at the long-term performance of WEO forecasts for an extended data set on the G-7 countries. Section XI compares the WEO forecasts to the Consensus values, while Section XII analyzes the potential gains from combining these two sets of forecasts. Section XIII looks at recommendations for modifications to the WEO forecasts and forecasting procedures, and Section XIV concludes.

II. Description of Weo Dataset

A. Data Coverage

To assess the forecasting performance, we make use of the fact that four sets of short-term forecasts are available for the same variable since the WEO publishes both April and September current- and next-year forecasts. For example, four forecasts of GDP growth in the year 2000 are reported, namely the April and September 1999 next-year forecasts and the April and September current-year forecasts. Access to different forecast vintages allows us to address issues such as whether (and by how much) the error in the forecast gets reduced as the time towards the target date is shrunk. It also allows us to test another efficiency property embedded in an optimal forecast, namely that forecast revisions should themselves be unpredictable. In some cases we find evidence of significant biases in revisions, suggesting simple ways of improving upon the forecasts.

The WEO data set contains information on 178 countries over the period 1990–2003. These countries are collected into seven groups or regions, namely Africa (50 countries), Central and Eastern Europe (15), CIS and Mongolia (13), Developing Asia (24), Middle East (14), Western Hemisphere (33), and Advanced Economies (29). Data availability and data quality vary significantly across regions and there can be significant differences even within each region. Data quality and the extent to which outliers affect the results also depends on the type of variable being analyzed.

There are five variables for which forecasts are available, namely GDP growth, export volume, import volume, inflation, and current account balance in US dollars or as a percentage of the underlying economies’ GDP. Our analysis focuses on current and next-year forecasts. Both can be considered short-term forecasts. Longer-term forecasts are not further pursued in the analysis because of the rather short data sample which is unlikely to make a statistical analysis of long-term forecasting performance particularly informative.

B. Timing Conventions

Since the target variables are subject to data revisions, a choice has to be made concerning which data vintage to use to measure realized values or outcomes. To this end we follow common practice and use the first-available data in the April WEO issue of year t + 1 to measure the outcome of the predicted variable in period t (labeled yt) while next-year forecasts for period t + 1 are compared to the realized values for year t + 1 (yt+ 1) reported in the September WEO issue of year t + 2.

We will also make use in the analysis of the fact that we have both April and September forecasts of same-year and next-year realizations. This means that we have two sets of current-year forecasts generated in April and September, y^t,t,Apr, y^t,t,Sep, and two sets of next-year forecasts generated during the same months and labeled y^t+1,t,Apr, y^t+1,t,Sep In this notation the first subscript indicates the period being predicted while the second subscript indicates the year when the forecast was generated. The superscript indicates the month of the WEO issue where the WEO forecast was reported. This convention gives rise to four separate forecast errors:

et,tApr=yty^t,tAprApril current year forecast error
et,tSep=yty^t,tSepSeptember current year forecast error
et+1,tApr=yty^t+1,tAprApril current year next forecast error
et+1,tSep=yty^t+1,tSepSeptember next year forecast error

In addition we will also consider current year and next year forecast revisions defined as3

revt,t=y^t,tSepy^t,tAprrevision to the current year forecast
revt+1,t=y^t+1,tSepy^t+1,tAprrevision to the next year forecast.

Table 2 presents basic information on data coverage within each of the regions for the five variables of interest. A maximum of 14 annual data points (1990–2003) is available (13 for next-year forecasts that begin in 1991). The third column reports the number of observations (averaged across countries within each region) that the forecasting analysis makes use of after deleting missing observations and cases where the forecast is exactly identical to the realized value. This leads to a significant trimming of data in some regions. For example, at least eight September current year forecasts are available only for 41 out of 50 African countries and only 11 of the 24 Developing Asian economies had more than eight data points for this variable. Fortunately data on April and September next-year forecasts tend to be more complete although again there are some countries with incomplete data. Measured by data coverage, the data set is most complete for the Advanced Economies and least complete for CIS and Mongolia. Although data coverage does not vary a great deal across variables, the current account data tends to contain somewhat fewer observations.

Table 1.Sample Size, by Variable and Region
Number of CountriesIncludedAverage Number of

Observations per

Country
Real GDP
April current-year forecast errors
Africa504812.52
Central and Eastern Europe151510.80
CIS and Mongolia131310.15
Developing Asia242011.60
Middle East141311.23
Western Hemisphere333211.53
Advanced economies292913.59
September current-year forecast errors
Africa504111.20
Central and Eastern Europe151311.23
CIS and Mongolia131210.17
Developing Asia241111.73
Middle East141010.40
Western Hemisphere332110.86
Advanced economies292912.76
Export Volume
April current-year forecast errors
Africa504712.19
Central Eastern Europe151111.27
CIS and Mongolia13119.73
Developing Asia241611.63
Middle East141411.79
Western Hemisphere333110.77
Advanced economies292813.64
September current-year forecast errors
Africa504913.37
Central and Eastern Europe151410.86
CIS and Mongolia131210.67
Developing Asia242413.08
Middle East141413.07
Western Hemisphere333313.00
Advanced economies292813.57
Import Volume
April current-year forecast errors
Africa504712.21
Central and Eastern Europe151111.27
CIS and Mongolia13119.64
Developing Asia241711.53
Middle East141411.43
Western Hemisphere333110.81
Advanced economies292812.79
September current-year forecast errors
Africa504110.46
Central and Eastern Europe15810.25
CIS and Mongolia1389.75
Developing Asia24812.88
Middle East141011.00
Western Hemisphere331410.50
Advanced economies292812.79
CPI Inflation
April current-year forecast errors
Africa504811.81
Cent. Eastern Europe151511.20
CIS and Mongolia131310.77
Developing Asia242011.35
Middle East141310.46
Western Hemisphere333210.94
Advanced economies292913.24
September current-year forecast errors
Africa505013.48
Cent. Eastern Europe151511.67
CIS and Mongolia131310.77
Developing Asia242412.67
Middle East141413.29
Western Hemisphere333313.36
Advanced economies292913.62
Current Account
April current-year forecast errors
Africa504712.13
Cent. Eastern Europe151510.60
CIS and Mongolia131310.92
Developing Asia241910.68
Middle East141111.73
Western Hemisphere333111.00
Advanced economies292812.50
September current-year forecast errors
Africa504310.51
Cent. Eastern Europe151410.43
CIS and Mongolia131310.62
Developing Asia241111.00
Middle East14810.25
Western Hemisphere331710.29
Advanced economies292812.25
Source: Author’s calculations.
Source: Author’s calculations.
Table 2.Descriptive Statistics for Forecast Errors, by Variable and Region1(Averages across countries in region)
MeanMedianStandard

Deviation
Serial

Correlation
Fraction of

Positive Errors
Real GDP (annual change in percent)
April current-year forecast errors
Africa-1.17-0.813.190.210.34
Central and Eastern Europe-1.17-0.713.490.370.46
CIS and Mongolia-1.93-1.488.280.310.53
Developing Asia-0.38-0.332.220.250.49
Middle East-1.660.206.380.370.53
Western Hemisphere-0.64-0.612.410.230.39
Advanced economies-0.04-0.141.360.210.48
September current-year forecast errors
Africa-0.60-0.522.810.240.40
Central and Eastern Europe-0.110.112.370.330.56
CIS and Mongolia-1.05-0.566.350.270.61
Developing Asia0.160.241.240.270.57
Middle East0.670.223.670.350.58
Western Hemisphere-0.26-0.122.020.240.46
Advanced economies0.09-0.020.810.220.55
April next-year forecast errors
Africa-1.45-1.374.070.280.33
Central and Eastern Europe-1.63-0.923.900.370.39
CIS and Mongolia-2.17-1.938.400.480.51
Developing Asia-0.63-0.672.860.330.45
Middle East-1.060.116.630.320.49
Western Hemisphere-1.33-1.343.080.260.33
Advanced economies-0.55-0.742.060.290.42
September next-year forecast errors
Africa-1.48-1.414.020.230.33
Central and Eastern Europe-1.40-0.973.760.340.41
CIS and Mongolia-2.39-2.789.600.460.52
Developing Asia-0.53-0.682.840.310.45
Middle East-1.340.066.150.310.53
Western Hemisphere-1.16-1.162.960.240.35
Advanced economies-0.36-0.481.970.240.44
Current-year forecast revision
Africa-0.83-0.542.000.230.36
Central and Eastern Europe-0.94-0.452.360.260.51
CIS and Mongolia-1.02-0.855.420.420.53
Developing Asia-0.46-0.511.860.290.45
Middle East-2.33-0.626.170.210.44
Western Hemisphere-0.34-0.271.560.230.38
Advanced economies-0.11-0.070.990.260.49
Next-year forecast revision
Africa-0.08-0.071.550.280.45
Central and Eastern Europe-0.29-0.240.990.310.45
CIS and Mongolia-0.37-0.362.190.240.47
Developing Asia-0.20-0.221.250.280.45
Middle East1.100.274.690.460.47
Western Hemisphere-0.47-0.441.220.310.35
Advanced economies-0.20-0.220.710.210.39
Export Volume (annual change in percent)
April current-year forecast errors
Africa-0.30-1.0520.340.250.43
Central and Eastern Europe-1.32-1.1915.670.290.55
CIS and Mongolia-1.74-0.5533.940.300.49
Developing Asia0.700.688.950.220.51
Middle East-43.59-2.06110.920.260.47
Western Hemisphere-1.70-0.988.510.240.44
Advanced economies-0.39-0.584.830.260.47
September current-year forecast errors
Africa-2.05-0.6315.330.180.44
Central and Eastern Europe0.520.8311.510.370.59
CIS and Mongolia-1.530.0124.540.290.53
Developing Asia0.15-0.025.940.210.52
Middle East0.040.708.570.210.52
Western Hemisphere-0.77-0.506.290.170.51
Advanced economies0.220.213.180.180.55
April next-year forecast errors
Africa-0.16-1.3216.450.240.47
Central and Eastern Europe0.140.7111.800.220.56
CIS and Mongolia-15.76-6.0555.080.340.44
Developing Asia1.842.3111.470.250.56
Middle East-52.750.97121.390.240.51
Western Hemisphere-1.17-0.439.580.240.45
Advanced economies-0.32-0.685.580.240.46
September next-year forecast errors
Africa-0.49-2.0315.210.230.46
Central and Eastern Europe0.160.4811.820.240.55
CIS and Mongolia-6.19-4.0023.320.220.46
Developing Asia0.760.1110.250.270.52
Middle East-27.73-0.22106.900.210.53
Western Hemisphere-1.70-0.908.600.230.43
Advanced economies-0.29-0.325.540.220.49
Current-year forecast revision
Africa1.530.1615.120.190.49
Central and Eastern Europe-1.19-0.8410.270.320.49
CIS and Mongolia-3.41-0.9722.260.290.51
Developing Asia-0.150.014.820.150.52
Middle East-39.210.07104.990.270.46
Western Hemisphere-0.53-0.224.330.170.46
Advanced economies-0.48-0.543.390.270.41
Next-year forecast revision
Africa0.690.378.150.180.51
Central and Eastern Europe0.410.092.960.270.54
CIS and Mongolia-7.72-1.0448.640.230.48
Developing Asia0.230.062.560.190.51
Middle East-20.320.6077.690.220.51
Western Hemisphere0.470.374.470.200.55
Advanced economies-0.07-0.171.780.210.48
Import Volume (annual change in percent)
April current-year forecast errors
Africa-0.610.2515.290.220.46
Central and Eastern Europe-2.47-1.0720.290.380.49
CIS and Mongolia-1.15-1.6036.530.310.50
Developing Asia-0.150.108.680.310.53
Middle East-42.400.63138.340.330.49
Western Hemisphere0.990.9610.770.300.52
Advanced economies-0.29-0.465.390.230.48
September current-year forecast errors
Africa-2.47-3.3014.610.270.43
Central and Eastern Europe-0.64-0.6817.720.400.58
CIS and Mongolia0.632.8936.330.310.50
Developing Asia-0.51-0.537.340.310.47
Middle East1.732.0819.850.320.54
Western Hemisphere0.631.339.910.370.53
Advanced economies0.280.333.920.210.53
April next-year forecast errors
Africa0.09-0.2515.220.220.47
Central and Eastern Europe-0.311.5117.300.270.54
CIS and Mongolia-4.00-3.0723.620.300.48
Developing Asia0.741.0711.300.180.56
Middle East-56.451.62136.470.160.52
Western Hemisphere0.960.1212.440.300.52
Advanced economies-0.13-0.726.450.220.50
September next-year forecast errors
Africa0.05-0.1014.150.210.47
Central and Eastern Europe2.281.6413.210.270.58
CIS and Mongolia-1.52-1.1219.350.350.48
Developing Asia0.340.5610.990.150.55
Middle East-37.350.39122.320.240.50
Western Hemisphere1.021.2511.880.280.51
Advanced economies-0.02-0.286.270.210.51
Current-year forecast revision
Africa0.871.1014.690.230.50
Central and Eastern Europe-2.64-2.9911.490.370.41
CIS and Mongolia0.99-0.8517.070.230.53
Developing Asia-0.45-0.356.460.240.47
Middle East-73.03-0.58209.850.220.55
Western Hemisphere0.900.298.960.260.55
Advanced economies-0.47-0.623.740.210.47
Next-year forecast revision
Africa0.870.399.090.260.51
Central and Eastern Europe-1.880.0610.320.330.60
CIS and Mongolia-2.020.5320.780.230.65
Developing Asia0.510.114.330.300.56
Middle East-30.49-0.65115.140.320.57
Western Hemisphere0.690.057.180.270.47
Advanced economies-0.18-0.162.110.220.44
Inflation (in percent per year)
April current-year forecast errors
Africa57.480.60162.180.290.57
Central and Eastern Europe7.362.3024.370.390.53
CIS and Mongolia340.63126.95978.650.690.49
Developing Asia1.721.068.200.330.53
Middle East-2.03-0.868.490.280.34
Western Hemisphere18.691.7850.400.380.56
Advanced economies-0.08-0.030.940.200.44
September current-year forecast errors
Africa39.700.12133.390.230.54
Central and Eastern Europe2.030.178.790.270.47
CIS and Mongolia163.1664.00554.430.600.42
Developing Asia1.000.095.970.230.47
Middle East-0.59-0.418.610.190.43
Western Hemisphere7.060.5823.870.270.50
Advanced economies-0.09-0.050.490.230.42
April next-year forecast errors
Africa81.722.50177.600.260.66
Central and Eastern Europe16.054.0334.060.410.60
CIS and Mongolia229.71177.67592.490.830.70
Developing Asia1.451.289.160.420.55
Middle East-0.77-0.9611.320.380.38
Western Hemisphere10.852.6662.080.380.59
Advanced economies-0.12-0.131.430.360.43
September next-year forecast errors
Africa74.941.80164.020.260.62
Central and Eastern Europe16.043.2732.430.390.58
CIS and Mongolia190.99153.65590.620.740.65
Developing Asia1.630.629.150.360.52
Middle East-1.77-1.1510.960.270.32
Western Hemisphere7.811.6054.870.290.58
Advanced economies-0.19-0.131.200.330.40
Current-year forecast revision
Africa17.030.2738.110.220.56
Central and Eastern Europe5.001.7819.150.200.58
CIS and Mongolia182.11120.45551.080.610.56
Developing Asia0.310.153.210.290.52
Middle East0.19-0.013.180.340.51
Western Hemisphere9.670.2929.170.180.51
Advanced economies0.030.030.760.190.51
Next-year forecast revision
Africa3.920.3011.010.210.57
Central and Eastern Europe1.600.697.300.150.63
CIS and Mongolia36.8311.26112.550.420.69
Developing Asia0.220.222.780.190.56
Middle East0.740.403.840.340.57
Western Hemisphere3.200.0712.270.180.57
Advanced economies0.050.010.700.260.53
Current Account (in percent of GDP)
April current-year forecast errors
Africa-0.210.355.700.270.52
Central and Eastern Europe20.950.5069.310.300.48
CIS and Mongolia-7.490.0395.360.240.61
Developing Asia0.580.425.780.280.58
Middle East-1.36-0.2412.980.230.61
Western Hemisphere-0.61-0.184.170.250.46
Advanced economies0.250.101.780.260.52
September current-year forecast errors
Africa0.170.255.250.250.54
Central and Eastern Europe4.300.5814.560.270.52
CIS and Mongolia-21.04-1.4782.000.390.54
Developing Asia1.280.604.690.280.63
Middle East-0.53-0.296.160.270.56
Western Hemisphere0.17-0.122.670.290.51
Advanced economies0.030.051.330.260.51
April next-year forecast errors
Africa-1.04-0.216.730.280.46
Central and Eastern Europe34.91-0.19110.890.390.41
CIS and Mongolia40.181.50128.560.340.59
Developing Asia0.780.677.640.400.56
Middle East0.301.8218.520.310.65
Western Hemisphere-1.52-0.835.360.300.39
Advanced economies0.390.092.420.380.52
September next-year forecast errors
Africa-0.670.206.880.240.50
Central and Eastern Europe8.50-0.7732.260.330.46
CIS and Mongolia1.510.6211.530.390.57
Developing Asia1.110.287.160.340.58
Middle East-1.691.0417.850.250.59
Western Hemisphere-0.93-0.504.990.270.42
Advanced economies0.220.092.250.350.51
Current-year forecast revision
Africa-0.29-0.254.690.220.50
Central and Eastern Europe18.030.1862.400.260.44
CIS and Mongolia16.210.9962.230.350.64
Developing Asia0.100.192.750.240.56
Middle East1.581.125.370.220.66
Western Hemisphere0.130.101.730.280.48
Advanced economies0.220.131.180.260.56
Next-year forecast revision
Africa-0.31-0.404.600.270.48
Central and Eastern Europe29.28-0.0499.550.220.39
CIS and Mongolia42.120.26135.870.330.64
Developing Asia-0.010.082.320.260.58
Middle East2.172.245.320.240.69
Western Hemisphere0.150.041.670.220.52
Advanced economies0.170.151.230.220.52

Except for the median which is the median of the mean forecast error across countries.

Except for the median which is the median of the mean forecast error across countries.

III. Properties of Optimal Forecasts

To evaluate the quality of the WEO forecasts it is necessary to establish a set of testable properties that an optimal forecast should have. In this section we discuss the nature of such properties. In all cases the properties are established under the assumption that the objective function is of the mean squared error (MSE) type so the forecasts minimize a symmetric, quadratic loss function. Different properties hold for other loss functions. For a theoretical discussion and derivation of these properties, see Patton and Timmermann (2004).

A. Unbiasedness and Lack of Serial Correlation

Most fundamentally, an optimal forecast should be unbiased and serially uncorrelated. Define the generic forecast errors for period t or t + 1, computed at time τ, as

To test the basic unbiasedness and uncorrelatedness properties, one can perform simple regressions

For an efficient forecast we must have α = 0 (unbiasedness) in (1) and α = 0, β = 0 in (2), implying unbiasedness and absence of serial correlation. The first regression gives rise to a simple student-t test of α = 0, while the second leads to an F-test. Adding the forecast, ŷt+ 1, τ, to both sides of equation (2), this regression is easily seen to be equivalent to the conventional Mincer-Zarnowitz (1969) levels regression

In this regression unbiasedness of the forecast translates into a requirement that α = 0, β = 1.

B. Efficiency Properties More Generally

Unbiasedness and absence of serial correlation in the forecast errors can be thought of as weak efficiency requirements. A much more general and stricter orthogonality condition holds for optimal forecasts under MSE loss. Since an optimal forecast should be the conditional expectation of the predicted variable of interest, if the forecaster uses all available information efficiently, then no variable in the current information set should be able to predict future forecast errors. To test this, let zt be any such variable in the forecaster’s information set at time t, Ωt. An implication of informational efficiency is that α = β = 0 in the regression

where εt+1 is a serially uncorrelated, zero-mean error term. The relationship between unbiasedness and absence of serial correlation on the one hand (equation (2)) and informational efficiency according to (4) more generally is similar to the relationship between the weak and semi-strong versions of the market efficiency hypothesis. According to the weakly efficient hypothesis, past values of the variable itself should not help predict future values. The semi-strong version tightens this restriction by requiring that no publicly available information should help forecast future values.

C. Forecast Revisions: Efficiency Tests Without Measurement Problems

Forecast revisions are of fundamental interest in a forecast evaluation exercise for one simple reason: If a sequence of forecasts is optimal, then the forecast revisions should themselves be unpredictable (technically a martingale difference sequence).

Indeed, if this were not the case and, say, forecast revisions between April and September were themselves predictable, then the original (April) forecast would not be optimal. Suppose, for example, that it is known that on average the September forecast of next-year output growth tends to be ¼ of 1 percent higher than the April forecast. Then the April forecast should be revised upwards by this amount to reflect the better information available in September of each year.

Another advantage of studying revisions is that predictable patterns in revisions, if detected, automatically tells the forecaster how to improve the original forecast, namely by amending it by the fitted value of the forecast revision. Hence if the April forecast of the revision in the forecast between April and September is

then the original April forecast, y^t,tApr can be replaced by an improved forecast, y˜t,tApr, as follows:

More generally, if ΩtSep is the forecaster’s information set in September, ΩtApr is the information set in April—which is a sub-set of the September information set, ΩtAprΩtSep — and if forecasts are formed optimally as conditional expectations, i.e.

then by the law of iterated expectations E[y^t+1,tSep|ΩtApr]=y^t+1,tApr and so the revision, defined as revt+1,t=y^t+1,tSepy^t+1,tApr must be zero-mean:

A similar result holds for the current-year revisions, revt,t=y^t,tSepy^t,tApr:

Notice, however, that in general E[revt+1,t|ΩtSep]0 and E[revt,t|ΩtSep]0 provided that any new information of use to the forecaster arrives between April and September of year t.4 It is worth pointing out that we ignore the effect of estimation errors, which can induce serial correlation in the forecast errors even if the forecaster knows the true model. This is akin to learning effects—see Timmermann (1993) for a discussion of this point in the context of predictability of financial returns.

An important implication follows from these simple results: Forecast optimality can be tested without having data on the target variable, y. This is important since, given the availability of different vintages of the target variable, it is not clear whether the forecasts should be compared to first-issue, second (revised) or the “final” data revision. This matters considerably in practice as witnessed by the recent literature on “real-time” macroeconomic data, (see Croushore, 2005). By analyzing data revisions we can effectively construct a test that is not sensitive to how well the underlying data is being measured.

D. Non-Increasing Variance of Forecast Errors as Forecast Horizon Is Decreased

A final property that an optimal forecast should have is that the variance of the forecast error should decline as more information becomes available. This means that the April current year (next year) forecast errors should have a greater variance than the September current-year (next year) forecast errors:

Intuitively this simply reflects that more information about the outcome in the current or next year is known in September than in April of the same year. This can be formally tested through a variance ratio test or (more appropriately given the small sample size here) by considering patterns in the variance of forecast errors associated with different forecast horizons.

IV. Empirical Results

With the dataset and benchmark properties of an optimal forecast in place, we proceed to analyze the empirical evidence. Table 3 reports summary statistics for the forecast errors and forecast revisions grouped by variable and region. We show the mean, median, and standard deviation of the forecast error, the average absolute value of the coefficient of first-order serial correlation in the forecast errors and the percentage of positive values of the forecast error. In all cases these statistics are computed based on the cross-section of countries within a particular region. For example, the median value is the median of the mean values across countries in a given region, while the standard deviation is computed across the mean values for countries in the region. In the sequel we discuss the main empirical findings.

Table 3.Tests for Biasedness and Serial Correlation of Forecast Errors(Share of countries in region with significant test statistics)
Forecast Error Bias (α^)1/Serial Correlation (β^)3/
T-Value for α^Fraction of

Bootstrap

P-Value

<0.052/
|tβ^|>2Fraction of

Bootstrap

P-Value

<0.054/
Percent of

Significant

Sign Tests 5/

(P-Value <.05)
tα^<2tα^>2
Real GDP
April current-year forecast errors
Africa0.400.000.250.060.150.17
Central and eastern Europe0.130.000.070.130.130.20
CIS and Mongolia0.080.080.150.080.000.15
Developing Asia0.100.050.050.000.000.15
Middle East0.000.080.080.310.150.08
Western Hemisphere0.190.000.130.030.060.03
Advanced economies0.030.030.070.000.030.10
September current-year forecast errors
Africa0.200.000.150.120.150.15
Central and eastern Europe0.000.080.000.150.000.08
CIS and Mongolia0.000.170.080.170.080.17
Developing Asia0.000.090.000.000.000.18
Middle East0.000.200.200.300.200.10
Western Hemisphere0.100.050.050.050.000.05
Advanced economies0.000.100.070.070.100.17
April next-year forecast errors
Africa0.380.040.330.150.330.35
Central and eastern Europe0.270.000.070.270.270.07
CIS and Mongolia0.150.000.000.460.080.08
Developing Asia0.170.040.090.300.130.09
Middle East0.000.070.000.140.140.07
Western Hemisphere0.360.000.270.090.150.18
Advanced economies0.280.030.240.140.210.14
September next-year forecast errors
Africa0.400.020.330.130.290.33
Central and eastern Europe0.200.000.070.200.200.13
CIS and Mongolia0.080.000.000.380.150.08
Developing Asia0.040.040.040.170.130.09
Middle East0.000.070.070.140.210.07
Western Hemisphere0.330.000.240.060.210.09
Advanced economies0.240.070.210.000.140.10
Current-year forecast revision
Africa0.290.000.170.100.100.12
Central and eastern Europe0.180.000.000.000.000.09
CIS and Mongolia0.000.000.000.200.000.00
Developing Asia0.080.000.080.080.080.08
Middle East0.000.000.000.100.000.00
Western Hemisphere0.200.000.130.000.000.00
Advanced economies0.040.040.040.070.040.11
Next-year forecast revision
Africa0.070.000.020.100.050.07
Central and eastern Europe0.220.000.000.220.110.00
CIS and Mongolia0.000.000.000.000.000.00
Developing Asia0.080.000.080.000.000.00
Middle East0.000.000.000.380.000.00
Western Hemisphere0.200.000.130.070.130.07
Advanced economies0.190.000.070.070.070.11
Export Volume
April current-year forecast errors
Africa0.090.000.090.040.060.06
Central and eastern Europe0.000.090.000.180.000.09
CIS and Mongolia0.000.000.000.180.090.00
Developing Asia0.000.060.060.000.000.06
Middle East0.070.000.000.140.070.07
Western Hemisphere0.060.000.030.030.030.06
Advanced economies0.000.040.040.140.040.04
September current-year forecast errors
Africa0.100.000.040.040.020.04
Central and eastern Europe0.000.070.070.210.070.14
CIS and Mongolia0.000.000.000.170.000.08
Developing Asia0.000.040.000.040.040.08
Middle East0.000.070.000.070.000.07
Western Hemisphere0.000.000.000.030.030.09
Advanced economies0.000.040.000.070.040.11
April next-year forecast errors
Africa0.070.020.020.020.000.04
Central and eastern Europe0.000.000.000.000.000.23
CIS and Mongolia0.080.000.000.150.000.08
Developing Asia0.000.090.000.050.000.18
Middle East0.000.070.070.000.000.14
Western Hemisphere0.030.000.000.030.000.00
Advanced economies0.040.070.040.040.040.07
September next-year forecast errors
Africa0.040.020.040.000.020.06
Central and eastern Europe0.000.000.000.080.000.17
CIS and Mongolia0.080.000.000.000.000.08
Developing Asia0.000.040.040.130.000.08
Middle East0.000.140.140.000.000.07
Western Hemisphere0.060.000.060.030.030.00
Advanced economies0.040.040.000.070.040.07
Current-year forecast revision
Africa0.040.000.000.020.020.10
Central and eastern Europe0.000.000.000.140.070.14
CIS and Mongolia0.000.000.000.230.150.15
Developing Asia0.040.000.000.000.000.08
Middle East0.000.000.000.070.000.00
Western Hemisphere0.090.000.000.060.030.06
Advanced economies0.070.000.040.110.000.07
Next-year forecast revision
Africa0.040.020.020.040.060.04
Central and eastern Europe0.000.000.000.070.000.00
CIS and Mongolia0.000.080.000.080.000.08
Developing Asia0.000.040.000.000.000.00
Middle East0.000.070.000.070.000.14
Western Hemisphere0.000.030.000.060.030.03
Advanced economies0.070.040.070.070.070.04
Import Volume
April current-year forecast errors
Africa0.040.020.020.040.060.06
Central and eastern Europe0.000.000.000.360.090.00
CIS and Mongolia0.000.000.000.180.000.09
Developing Asia0.000.060.000.120.120.12
Middle East0.000.000.000.290.140.00
Western Hemisphere0.030.000.000.030.030.00
Advanced economies0.000.000.000.040.000.04
September current-year forecast errors
Africa0.070.000.070.070.070.07
Central and eastern Europe0.000.000.000.130.130.13
CIS and Mongolia0.000.000.000.130.000.00
Developing Asia0.000.000.000.000.000.00
Middle East0.000.000.000.100.100.00
Western Hemisphere0.000.000.000.140.140.00
Advanced economies0.000.000.000.040.000.07
April next-year forecast errors
Africa0.040.040.020.020.020.04
Central and eastern Europe0.000.080.000.000.000.00
CIS and Mongolia0.080.000.000.000.080.08
Developing Asia0.000.000.000.090.000.09
Middle East0.070.000.000.000.000.14
Western Hemisphere0.000.030.000.090.030.06
Advanced economies0.000.000.000.040.040.07
September next-year forecast errors
Africa0.020.040.040.090.020.09
Central and eastern Europe0.000.100.100.100.000.10
CIS and Mongolia0.090.000.000.090.000.00
Developing Asia0.000.000.000.050.000.09
Middle East0.070.000.000.000.000.07
Western Hemisphere0.000.030.030.060.060.03
Advanced economies0.000.000.000.040.000.07
Current-year forecast revision
Africa0.030.030.030.050.050.07
Central and eastern Europe0.000.000.000.110.000.00
CIS and Mongolia0.000.000.000.000.000.14
Developing Asia0.000.000.000.180.090.00
Middle East0.000.000.000.110.000.00
Western Hemisphere0.000.000.000.000.000.08
Advanced economies0.140.070.180.000.110.14
Next-year forecast revision
Africa0.030.050.000.030.000.10
Central and eastern Europe0.000.000.000.220.000.00
CIS and Mongolia0.000.000.000.140.000.14
Developing Asia0.000.000.000.000.000.00
Middle East0.000.000.000.110.000.00
Western Hemisphere0.000.000.000.090.000.00
Advanced economies0.180.000.140.070.070.07
Inflation
April current-year forecast errors
Africa0.000.190.130.060.130.21
Central and eastern Europe0.000.070.000.330.070.13
CIS and Mongolia0.000.080.000.690.310.00
Developing Asia0.000.050.000.150.000.15
Middle East0.080.000.000.150.000.15
Western Hemisphere0.030.190.130.220.090.16
Advanced economies0.070.030.000.070.070.07
September current-year forecast errors
Africa0.040.080.000.060.040.16
Central and eastern Europe0.070.000.000.200.000.00
CIS and Mongolia0.000.080.000.620.150.00
Developing Asia0.080.000.080.040.080.13
Middle East0.000.000.000.000.000.07
Western Hemisphere0.030.060.030.120.060.06
Advanced economies0.170.000.030.070.030.00
April next-year forecast errors
Africa0.000.250.190.190.170.31
Central and eastern Europe0.000.200.130.330.200.33
CIS and Mongolia0.000.310.000.920.310.31
Developing Asia0.000.170.130.350.260.26
Middle East0.290.140.140.290.210.36
Western Hemisphere0.150.270.240.330.390.36
Advanced economies0.140.030.140.100.170.10
September next-year forecast errors
Africa0.020.220.160.140.140.34
Central and eastern Europe0.000.270.200.330.200.40
CIS and Mongolia0.000.230.080.770.150.23
Developing Asia0.130.130.080.210.170.25
Middle East0.210.140.140.000.140.21
Western Hemisphere0.120.240.270.150.360.42
Advanced economies0.070.000.070.170.140.07
Current-year forecast revision
Africa0.000.140.020.080.020.18
Central and eastern Europe0.000.070.000.200.000.07
CIS and Mongolia0.000.080.000.620.310.15
Developing Asia0.000.080.040.080.040.17
Middle East0.000.000.000.290.000.21
Western Hemisphere0.000.120.000.060.000.12
Advanced economies0.030.030.070.070.030.07
Next-year forecast revision
Africa0.020.080.040.060.060.18
Central and eastern Europe0.000.130.070.000.000.20
CIS and Mongolia0.000.000.000.380.080.38
Developing Asia0.000.170.040.090.090.13
Middle East0.000.000.000.210.070.07
Western Hemisphere0.000.060.030.060.030.24
Advanced economies0.030.100.030.140.100.10
Current Account (in percent of GDP)
April current-year forecast errors
Africa0.020.020.020.130.020.04
Central and eastern Europe0.000.000.000.070.070.00
CIS and Mongolia0.000.080.000.080.000.08
Developing Asia0.000.160.110.050.050.16
Middle East0.000.090.000.090.090.18
Western Hemisphere0.030.000.030.100.030.06
Advanced economies0.000.070.070.070.040.07
September current-year forecast errors
Africa0.000.090.020.050.050.09
Central and eastern Europe0.000.000.000.070.000.07
CIS and Mongolia0.000.080.080.230.080.23
Developing Asia0.000.180.180.090.090.36
Middle East0.000.000.000.000.000.13
Western Hemisphere0.060.060.060.000.060.00
Advanced economies0.000.040.000.110.040.11
April next-year forecast errors
Africa0.130.020.130.040.090.11
Central and eastern Europe0.070.000.070.200.130.13
CIS and Mongolia0.000.230.230.150.150.46
Developing Asia0.000.130.090.260.260.17
Middle East0.000.140.140.140.140.36
Western Hemisphere0.240.030.270.090.240.12
Advanced economies0.040.180.140.320.180.14
September next-year forecast errors
Africa0.090.040.090.090.020.13
Central and eastern Europe0.070.000.000.130.130.07
CIS and Mongolia0.080.150.080.230.000.15
Developing Asia0.000.130.090.090.040.17
Middle East0.000.070.000.000.070.14
Western Hemisphere0.160.030.160.160.160.03
Advanced economies0.040.140.110.290.070.14
Current-year forecast revision
Africa0.020.040.070.040.000.09
Central and eastern Europe0.130.000.000.130.000.07
CIS and Mongolia0.000.080.000.170.080.42
Developing Asia0.000.080.000.080.080.15
Middle East0.000.180.000.000.000.09
Western Hemisphere0.000.000.000.000.000.06
Advanced economies0.000.040.040.070.040.04
Next-year forecast revision
Africa0.070.020.020.110.070.04
Central and eastern Europe0.070.000.000.070.000.07
CIS and Mongolia0.000.080.000.250.080.25
Developing Asia0.000.000.000.000.000.08
Middle East0.000.180.090.000.000.36
Western Hemisphere0.000.000.000.060.000.06
Advanced economies0.040.040.000.000.000.00
Source: Author’s calculations.

The bias coefficient α is defined in equation (1) in the main text.

Fraction of bootstrapped p-values for the null hypothesis α^=0 that are smaller than 0.05 in a two-sided test.

The serial correlation coefficient β is defined in equation (2).

Fraction of bootstrapped p-values for the F-test of the joint null hypothesis of α = 0 and β = 0 that are smaller than 0.05.

Fraction of significant test values (p-value of less than or equal to 0.05) for a test of the null hypothesis that the fraction of positive forecast errors equals 0.5.

Source: Author’s calculations.

The bias coefficient α is defined in equation (1) in the main text.

Fraction of bootstrapped p-values for the null hypothesis α^=0 that are smaller than 0.05 in a two-sided test.

The serial correlation coefficient β is defined in equation (2).

Fraction of bootstrapped p-values for the F-test of the joint null hypothesis of α = 0 and β = 0 that are smaller than 0.05.

Fraction of significant test values (p-value of less than or equal to 0.05) for a test of the null hypothesis that the fraction of positive forecast errors equals 0.5.

A. GDP Growth

Current-year forecasts

For the real GDP growth rate variable the mean of the current-year forecast error (i.e., the bias averaged across time and across countries) is very close to zero for the Advanced Economies. Biases in April current-year forecasts are much larger—exceeding more than one percent—and negative for Africa, Central and Eastern Europe, CIS and Mongolia, and the Middle East. As expected, this bias is reduced significantly in the September current-year forecasts. Although the April biases appear to be rather large, it should also be noted that they reflect some very large outliers whose values are predominantly negative and thus represent overpredictions. Indeed, the standard deviations of the April current-year forecast errors tend to be largest for those regions where the greatest biases were found—exceeding 8 percent for CIS and Mongolia and 6 percent for the Middle East.

Due to the presence of such outliers in the data we also consider more robust statistics such as the median forecast error and the proportion of positive forecast errors (underpredictions).

Provided that the underlying shocks are not drawn from asymmetric distributions, one would expect the median to be close to zero and the proportion of positive forecast errors to be close to 50 percent on average if the underlying forecasting model is not misspecified. Again the data reveals systematic problems for some of the regions: between 34 and 40 percent of the same-year forecasts for the African region are overpredictions of subsequent GDP growth (negative mean forecast errors). Consistent with this, the median forecast error remains large and negative (-0.81 for this region) as it does for Central and Eastern Europe and CIS and Mongolia.

Forecasts in all regions pass the test that the variance of the September forecast errors should be smaller than the variance of the April forecast errors of the same variable. Furthermore, in many regions the reduction in uncertainty between the April and September forecast appears to be quite large. For example, the average standard deviation of the current-year forecast error in the Advanced Economies is reduced from 1.36 percent in April to 0.81 percent in September, representing a 40 percent reduction.

Next-year forecasts

Biases in the next-year forecast errors generally exceed those observed in the current-year forecasts. Interestingly, in every single region the mean April or September biases are negative and this also holds for the median bias in all regions with exception of the Middle East. This suggests that the WEO in general overpredicts next-year GDP growth. Furthermore, whereas the average bias in the current-year predictions for the Advanced Economies is very small, it is quite sizeable in the next year forecast where it takes values of -0.36 and -0.55 percent, depending on the reporting date of the forecast. Estimates of the standard deviations of the forecast errors associated with the April and September next year forecasts are much more similar than their current year counterparts. This suggests that far less is learned between April and September about next-year growth than is learned between these months about growth in the current year.5

The proportion of positive next-year forecast errors is again very low for Africa (0.33) and the Western Hemisphere (0.35). The predominance of regions with proportions of positive signs below 0.5 is consistent with the tendency of the WEO forecasts to over-predict next-year GDP growth.

Serial correlation in the forecast errors appears to also be a problem in some regions. The fourth column of Table 3, which reports the average of the absolute value of the first-order autocorrelation in the forecast error is quite high in central and eastern Europe and CIS and Mongolia in particular.

Turning to the forecast revisions between the April and September WEO publications, which should have a mean of zero, there is systematic evidence of negative biases. This is consistent with the April and September forecasts both overpredicting GDP growth on average, but the April forecast being more optimistic than the September value (so the mean change is negative). Hence on average the September forecast is being revised downwards when compared to the April value. This finding is corroborated in the median values as well as in the proportion of positive forecast revisions (which consistently lies below one-half) and is information that could easily be used to improve upon the WEO growth forecasts.

Another feature that is worth noting in the forecast revisions is that the standard deviation of the revision is generally quite a bit larger for the current-year values than for the next-year values. Again, this reinforces the earlier observation that information arriving between April and September more strongly affects current-year than next-year forecasts.

B. Inflation

Very high inflation rates characterized a number of countries during the sample period so it is unsurprising that outliers tend to be very large for this variable and certainly larger than for real GDP growth. As a consequence we focus our analysis on the relatively robust measures of forecasting performance such as the proportion of positive forecast errors. For the current-year forecasts this does not deviate too strongly from 50 percent in any of the regions, except for the Middle East, where only between 34 and 43 percent of the April and September current-year forecast errors are positive, and to a lesser extent for the Advanced Economies where 43 percent of the signs are positive.

A rather different picture emerges for the next-year forecast errors. Between 60 and 70 percent of the April forecast errors are positive for Africa, central and eastern Europe and CIS and Mongolia. These proportions are closer to 60 percent for the September forecasts, but remain somewhat higher than 50 percent, indicating a tendency toward underprediction of inflation in these countries. Furthermore, all forecast revisions have positive means and more than 50 percent of the forecast revisions are positive. A particularly high percentage is observed among the next-year revisions for CIS and Mongolia and central and eastern Europe which generally see the average forecast revised upward. Hence there is a tendency for both the WEO’s current-year and next-year inflation forecasts to be raised between April and September. Since the September forecasts are generally more accurate than their April counterparts, this suggests that the April WEO inflation forecasts can be improved by increasing their value.

We also consider whether the standard deviation of the April forecast errors is greater than that of the September forecast errors. Although outliers make it difficult to interpret some of the values, this appears generally to be the case.

C. Export and Import Volumes

Data on export and import volume is also affected strongly by outliers. Outliers in export volume with values in triple digits affect Africa, CIS and Mongolia and the Middle Eastern region, whereas central and eastern Europe, developing Asia, Western Hemisphere and the Advanced Economies are not so strongly influenced by these. For the regions not affected by outliers, the bias appears rather modest—ranging between 0.22 and -0.39 for the Advanced Economies, for example. Furthermore, the September current-year and next-year forecast errors generally have a smaller standard deviation than the corresponding April values, suggesting that information arriving between April and September is used to improve upon the forecasts. For the vast majority of regions and forecast horizons the proportion of positive forecast errors (or revisions) is close to 50 percent as one would expect for a weakly efficient forecast under a symmetric error distribution.

For the import volume data only the Advanced Economies, Western Hemisphere and developing Asia are not affected by triple-digit outliers. Average biases are small in economic terms and at or below one percent in absolute value for these regions. In all regions we continue to see a proportion of positive forecast errors that is quite close to 50 percent.

D. Current Account Balance

Current account balance figures measured in U.S. dollar terms or as a percentage of GDP suggest a slight tendency for the WEO current-year forecasts to under predict, particularly in central and eastern Europe and developing Asia. Forecast revisions mostly have positive means so the September forecast is generally above the April value, again indicating a simple method for improving upon the quality of the April forecasts.

V. Analysis of Statistical Significance

Whether the biases documented in the previous table should be of concern depends on how systematic they are. This issue can best be addressed by undertaking a more in-depth statistical analysis. Such an analysis is of course tempered by the short data sample which potentially invalidates inference relying on asymptotic distributions but also lowers the power of a statistical analysis to detect misspecification in the forecasting models even when this is present. Again countries with fewer than eight observations will be excluded from the statistical analysis. Considerable caution should be exercised when interpreting the statistical inference results since the sample size used here is very small, and finite-sample distortions of standard test-statistics that correct for heteroskedasticity and autocorrelation in the regression residuals are well known (see, for example, Den Haan and Levin, 1997; Kiefer, Vogelsang, and Bunzel, 2000; and Kiefer and Vogelsang, 2005).

To deal with the problem that the small-sample properties of the simple t- and F-statistics are such that standard critical levels may not provide a reliable guide to inference, we designed a bootstrap experiment. This procedure repeatedly draws values of the forecast errors (e1, ...., eT) with replacement from the empirical distribution function to construct a sample whose length (T) is identical to that of the original data sample. Having constructed an artificial sample in this way (e1(b)b,.,eT(b)b), where b is an indicator for the bth bootstrap and 1(b), …, T(b) are randomly drawn integer values between 1 and T, we recalculate the test statistics of interest, e.g., t− and F −statistics associated with the efficiency regressions. We repeat this in 5,000 bootstrap experiments to construct a histogram for the distribution of the test statistic. The value of the test statistic found for the actual data is then compared with this bootstrapped distribution to get bootstrapped p-values. We shall report the proportion of countries for which the actual test statistic exceeds the 95th percentile of the bootstrapped distribution (using a two-sided test for the t-statistic).

Using equation (1) the first two columns of Table 4 report the proportion of included countries in the various regions for which the t-statistic associated with the mean forecast error is less than -2 or greater than 2, respectively. The third column reports the proportion of bootstrapped p-values for α = 0 that fall below 0.05 using a two-sided test. The fourth column reports the percentage of regressions for which the absolute value of the t-statistic of β in the weak efficiency regression (2) is greater than two. The fifth column reports the percentage of cases where the F-test for the joint hypothesis α = 0, β = 0 in (2) exceeds its 5 percent critical level, while the final column reports the percentage of significant values of a sign test for whether the proportion of positive forecast errors differs from one-half, again using a 5 percent critical level. The purpose of reporting so many test statistics is to get a broader picture of possible forecast inefficiencies and to account for the fact that the individual test statistics are surrounded by more than the usual uncertainty due to the very small samples entertained here. Caution should therefore be exercised when interpreting the results.

Table 4.Results for Program Countries
Number of ObservationsMean ErrorFraction of Negative Errors
GDP forecasts
April current-year forecast errors958-0.850.53
September current-year forecast errors958-0.290.43
April next-year forecast errors902-1.510.60
September next-year forecast errors902-1.370.59
Inflation
April current-year forecast errors9272.110.41
September current-year forecast errors9430.920.47
April next-year forecast errors8484.510.36
September next-year forecast errors8494.120.38
Source: Author’s calculations.
Source: Author’s calculations.

A. GDP Forecasts

First consider the April current-year forecasts. For close to 40 percent of the countries in the African region, the GDP growth forecasts were systematically too large.6 The bootstrapped test statistics confirm a significant bias for a much larger proportion of African countries—close to 30 percent—than should be expected if the forecasts were genuinely unbiased. This proportion is reduced to 15 percent when bias and serial correlation are jointly tested, most likely because of the weaker power of the joint test which requires estimation of an additional parameter. In fact we can only identify significant serial correlation for about 6 percent of the African countries (column four). Similarly, for around 15 percent of the African countries, the proportion of positive signs in the current year forecast errors is significantly different from one-half at the 5 percent critical level.

Between 10 and 20 percent of the countries in CIS and Mongolia and the Western Hemisphere also show evidence of a significant bias in the forecasts. The region where serial correlation in the forecast errors appears to be most important is the Middle East where 15 percent of the countries generate significant bootstrapped test statistics.

These findings mostly carry over to the September current-year forecasts. Forecast errors continue to be biased and serially correlated for around 15 percent of the countries in Africa and there is strong evidence of serial correlation for the Middle East and Africa. In contrast there is very little evidence that the current-year forecasts are biased or serially correlated in developing Asia or the developing Economies. Overall, the proportion of cases with a significant bias is lower in the September current-year forecasts compared to the April current-year forecasts.

Turning to the next year forecast errors, there is evidence of a significant upward bias in the forecasts for around 35 percent of the countries in Africa and almost 30 percent of the countries in the Western Hemisphere. Significant biases also affect more than 20 percent of the countries among the Advanced Economies. Serial correlation in next-year forecast errors plagues all regions, particularly Africa. All told, the bootstrapped p-values show a pattern of biased or serially correlated next-year forecast errors in all regions.

Current-year forecast revisions are biased for Africa and the Western Hemisphere but there is little evidence of serial correlation. Next-year forecast revisions are biased and serially correlated for more than 10 percent of the countries in the Western Hemisphere but otherwise the evidence against (weak) efficiency tends to be relatively mild.

Effect of IMF programs

A potential source of bias in the WEO forecasts is whether or not a country is engaged in one of the IMF programs. The bias could, as discussed by Mussa and Phillips (2002) work in either direction. Given the limited sample size available here, we consider the effect of any of the following types of programs, namely SBA (Stand By Arrangements), EFF (Extended Fund Facility), SAF (Structural Adjustment Facility), ESAF (Enhanced Structural Adjustment Facility) and PRGF (Power Reduction and Growth Facility). Our data comprises 350 program events, some of which lasted multiple years and were still continuing at the end of the sample. Discarding extreme outliers we ended up with between 850 and 960 program year observations to analyze.

The results, shown in Table 5, indicate the presence of systematic overpredictions (negative mean forecast errors) of GDP growth in program countries. The upward bias is smallest for the September current year forecasts and largest for the April next year forecast. A full 60 percent of the next year forecast errors are negative for the program countries.

Table 5.Predictability of Forecast Errors in Relation to Current Information Variables(Fraction of all countries in region with t-values for additional variables above or below indicated threshold)
US GDP GrowthGerman GDP

Growth
Oil PricesGlobal

Current Account

Discrepancy
<-2>2<-2>2<-2>2<-2>2
Real GDP
April current-year forecast errors
Africa0.020.040.000.060.040.040.060.06
Central/Eastern Europe0.000.330.000.070.000.130.000.20
CIS and Mongolia0.000.000.080.230.080.080.230.00
Developing Asia0.000.050.000.000.000.000.050.10
Middle East0.080.080.000.000.000.000.000.00
Western Hemisphere0.000.090.060.000.060.060.030.09
Advanced economies0.000.310.070.030.070.000.000.38
September current-year forecast errors
Africa0.000.050.050.070.000.120.070.02
Central/Eastern Europe0.000.080.000.000.000.150.080.15
CIS and Mongolia0.000.000.000.080.000.250.330.08
Developing Asia0.000.090.180.000.000.000.090.00
Middle East0.000.000.100.000.100.000.000.00
Western Hemisphere0.000.000.050.000.000.000.050.00
Advanced economies0.030.240.070.070.000.030.000.14
April next-year forecast errors
Africa0.000.000.020.020.020.130.060.02
Central/Eastern Europe0.000.000.070.000.130.070.000.07
CIS and Mongolia0.000.080.080.540.000.080.000.00
Developing Asia0.000.090.040.040.000.040.040.04
Middle East0.000.070.000.070.070.000.070.00
Western Hemisphere0.030.060.090.060.060.000.000.09
Advanced economies0.210.030.030.000.210.000.000.24
September next-year forecast errors
Africa0.020.000.020.040.020.060.060.06
Central/Eastern Europe0.000.000.070.000.070.000.070.00
CIS and Mongolia0.000.000.000.380.000.000.000.00
Developing Asia0.090.040.040.040.000.000.000.04
Middle East0.070.070.000.070.070.000.070.00
Western Hemisphere0.060.000.060.150.150.000.030.12
Advanced economies0.070.000.100.000.210.000.000.03
Current-year forecast revision
Africa0.050.050.050.020.070.000.050.05
Central/Eastern Europe0.000.360.000.090.000.000.000.00
CIS and Mongolia0.000.000.100.500.000.200.000.00
Developing Asia0.000.150.080.000.000.000.080.00
Middle East0.100.100.000.000.000.000.200.00
Western Hemisphere0.000.070.070.000.000.070.130.13
Advanced economies0.000.290.040.040.040.000.000.32
Next-year forecast revision
Africa0.070.020.020.070.020.020.020.05
Central/Eastern Europe0.000.000.110.000.000.110.220.00
CIS and Mongolia0.000.000.000.140.000.140.000.00
Developing Asia0.000.000.000.080.000.000.170.00
Middle East0.000.000.000.000.000.000.500.00
Western Hemisphere0.070.000.070.070.200.000.000.00
Advanced economies0.040.000.000.040.040.000.000.11
Inflation
April current-year forecast errors
Africa0.100.080.000.150.020.020.060.17
Central/Eastern Europe0.000.200.070.400.000.000.130.00
CIS and Mongolia0.000.080.000.690.000.000.150.15
Developing Asia0.000.050.050.300.000.050.000.00
Middle East0.150.000.150.000.000.000.000.15
Western Hemisphere0.030.250.000.220.060.030.030.03
Advanced economies0.000.140.070.030.000.070.210.03
September current-year forecast errors
Africa0.020.000.020.120.000.080.020.14
Central/Eastern Europe0.000.070.070.270.070.070.070.13
CIS and Mongolia0.000.000.080.620.000.000.540.08
Developing Asia0.000.080.080.080.080.040.040.04
Middle East0.070.070.210.000.070.000.000.07
Western Hemisphere0.000.180.000.180.000.000.060.00
Advanced economies0.030.030.000.100.000.140.070.00
April next-year forecast errors
Africa0.000.080.000.150.060.060.040.06
Central/Eastern Europe0.000.330.000.330.000.000.000.00
CIS and Mongolia0.000.230.000.920.000.000.000.00
Developing Asia0.130.300.170.260.000.040.090.09
Middle East0.140.140.070.000.000.000.000.07
Western Hemisphere0.000.300.000.270.060.120.090.06
Advanced economies0.070.170.240.140.030.000.000.00
September next-year forecast errors
Africa0.000.120.020.120.060.060.020.06
Central/Eastern Europe0.000.330.000.470.070.000.000.00
CIS and Mongolia0.000.230.000.770.000.000.080.08
Developing Asia0.080.250.170.250.000.040.130.00
Middle East0.070.070.140.000.000.000.000.07
Western Hemisphere0.000.270.000.270.060.060.150.06
Advanced economies0.070.170.210.100.000.030.000.00
Current-year forecast errors
Africa0.040.080.000.080.040.080.040.02
Central/Eastern Europe0.000.130.130.270.000.000.130.07
CIS and Mongolia0.000.080.000.540.000.000.000.31
Developing Asia0.000.080.000.040.130.080.000.04
Middle East0.070.070.070.070.000.070.000.00
Western Hemisphere0.000.060.000.060.030.090.000.03
Advanced economies0.000.100.100.070.000.030.070.03
Next-year forecast revision
Africa0.000.040.060.040.020.120.020.02
Central/Eastern Europe0.000.130.000.000.000.000.070.00
CIS and Mongolia0.000.150.000.230.000.000.000.23
Developing Asia0.000.040.090.040.040.040.000.04
Middle East0.000.000.070.140.000.210.070.00
Western Hemisphere0.000.000.150.030.060.120.000.03
Advanced economies0.000.070.000.100.000.030.000.00
Current Account (in percent of GDP)
April current-year forecast errors
Africa0.040.000.000.020.000.060.060.02
Central/Eastern Europe0.000.070.000.000.000.000.070.20
CIS and Mongolia0.000.000.000.000.000.000.310.08
Developing Asia0.160.000.000.050.000.050.050.00
Middle East0.090.000.000.090.000.090.090.00
Western Hemisphere0.060.000.000.060.060.060.030.06
Advanced economies0.040.040.110.040.070.040.180.00
September current-year forecast revision
Africa0.090.020.020.000.000.120.120.00
Central/Eastern Europe0.070.000.000.000.000.000.290.14
CIS and Mongolia0.080.000.000.080.000.080.230.23
Developing Asia0.090.090.000.000.000.000.090.00
Middle East0.130.250.000.250.130.000.130.00
Western Hemisphere0.000.180.060.120.000.060.000.00
Advanced economies0.070.040.070.040.110.070.110.00
April next-year forecast errors
Africa0.110.000.020.000.020.110.110.04
Central/Eastern Europe0.000.130.070.070.000.070.070.13
CIS and Mongolia0.150.000.000.080.080.080.080.08
Developing Asia0.090.000.000.000.000.170.130.00
Middle East0.070.140.000.210.070.000.000.07
Western Hemisphere0.090.060.030.090.000.090.000.06
Advanced economies0.070.140.110.070.000.140.140.07
September next-year forecast errors
Africa0.020.000.020.020.040.130.220.04
Central/Eastern Europe0.000.070.070.070.000.070.070.07
CIS and Mongolia0.380.000.000.000.150.000.000.15
Developing Asia0.040.040.040.090.000.170.130.00
Middle East0.000.210.070.070.140.000.000.07
Western Hemisphere0.000.090.000.090.030.090.030.06
Advanced economies0.110.070.140.070.000.070.110.04
Current-year forecast errors
Africa0.020.000.020.020.040.110.020.09
Central/Eastern Europe0.000.000.000.000.000.000.070.13
CIS and Mongolia0.080.000.000.080.000.000.170.08
Developing Asia0.000.000.000.000.000.000.080.08
Middle East0.000.000.000.000.090.000.000.00
Western Hemisphere0.130.000.000.130.000.130.060.06
Advanced economies0.110.210.070.000.070.000.000.00
Next-year forecast errors
Africa0.000.020.020.020.040.020.020.04
Central/Eastern Europe0.070.000.000.070.000.000.070.13
CIS and Mongolia0.000.000.000.000.000.000.000.08
Developing Asia0.000.000.000.000.000.000.000.00
Middle East0.090.000.090.000.000.090.000.00
Western Hemisphere0.190.000.000.130.000.190.000.06
Advanced economies0.070.040.070.040.070.040.040.00
Source: Author’s calculations.
Source: Author’s calculations.

Although the bias estimates appear large, it should be borne in mind that so were the average biases reported for countries in the same regions hosting most of the program countries (see Table 3).

B. Inflation Forecasts

As mentioned previously, the inflation data is affected by numerous outliers so we will not rely on standard test statistics and instead move directly to consider the bootstrap results. These reveal mild evidence of inefficiency in the current-year inflation forecasts. There appears to be some positive bias (underprediction of inflation) in the case of Africa and the Western Hemisphere. By far the strongest evidence against efficiency is found in the next-year forecast errors which reveal forecasts that are systematically downward-biased in most regions except for the Advanced Economies. However, forecast errors are serially correlated in the latter region so the null of no bias or serial correlation is rejected for around 15 percent of all countries (more than double the level expected under the null).

For the next year forecasts, with the exception of CIS and Mongolia, a greater-than-expected proportion of countries in the various regions generate a significant test statistic associated with the bias. The strongest evidence against efficiency comes from the serial correlation tests in column five, however, which shows that p-values below 5 percent were generated for between 15 percent and 40 percent of the countries in the various regions. In particular, more than 30 percent of the countries in the Western hemisphere show evidence of significant serial correlation in the forecast errors.

With few exceptions, forecast revisions reveal little systematic evidence of biases or serial correlation.

Effect of IMF programs

For the inflation forecasts again a large and systematic bias is observed for the countries enrolled in Fund-sponsored projects. However, the results in Table 5 show that the bias now goes in the opposite direction relative to what was observed for GDP growth as the inflation rate is underpredicted and the mean forecast error therefore takes a positive value. Again the largest bias—equal to 4.5 percent—is observed at the longest forecast horizon, i.e. for the April next year forecasts, while the mean bias is a more modest 0.9 percent for the September current year predictions.

C. Import and Export Volume, Current Account Balance

There is little evidence of systematic inefficiencies in the current-year or next year forecasts of trading volume. A similar conclusion holds for the current-year forecasts of the current account balance. However there does seem to be some evidence of bias and serial correlation in the next-year forecasts of the current account balance, where rejection rates are twice their expected values for many of the regions and sometimes four times as high (e.g., 24 percent in the case of the April next year forecasts for the Western Hemisphere or Developing Asia), often as a result of serial correlation.

VI. Can The WEO Forecast Errors Be Predicted?

The process whereby the WEO forecasts are generated puts considerable emphasis on integrating predictions across countries, regions and variables in order to produce a coherent and internally consistent projection of current and future economic activity. One way to analyze whether the procedures that are currently in place have their intended effect is to test for informational efficiency using a range of indicators of global economic activity. Such tests build on the moment condition E[et+1|Ωt] = 0, where Ωt is the forecaster’s information set at the time of the forecast (t), and are hence versions of the efficiency tests in (4).

In our empirical application we focus on four such predictor variables. First, we consider the WEO prediction of U.S. GDP growth. This is an obvious choice given the size and leading role that the U.S. economy plays in shaping global economic activity. The second instrument is the WEO prediction of German output growth—again motivated by the significance of this economy to regional and global growth.7 Finally, we also use the WEO forecast of oil prices and a global current account discrepancy instrument as predictors. Oil prices are an obvious choice since they are an important determinant of economic growth and inflation in a number of economies. The global current account discrepancy is constructed as the sum total of current accounts across all countries scaled by 15 global export. This figure should be equal to zero but may differ from this value due to measurement errors.

Table 6 shows the outcome of this exercise. Within each region and for each of the predictor variables the table reports the proportion of t-values below minus two and above two, respectively. Results indicative of a failure to fully account for the predicted U.S. GDP growth should show up in the form of a proportion of significant t-values somewhat higher than 5 percent. There is also information in the sign of the t-statistic. For GDP growth a higher proportion of positive and significant values than negative and significant t-statistics would reveal a failure to fully account for the spillover of U.S. GDP growth to other countries.

Table 6.Output Gaps and the Predictability of Forecast Errors in Advanced Economies(Value of t-statistics for the coefficient of the output gap in forecast efficiency regression)
Current-YearNext-YearForecast Revisions
AprilSeptemberAprilSeptemberCurrent -

Year
Next-

Year
Real GDP
Australia-1.640.91-0.030.91-2.69-1.91
Austria-1.56-1.41-1.09-1.34-1.10-0.24
Belgium-2.01-1.51-1.35-1.54-0.730.13
Canada-0.58-0.530.53-0.22-0.28-0.04
Cyprus0.020.020.000.00-0.440.76
Denmark-1.67-0.25-1.20-0.85-3.02-2.11
Finland-0.17-0.04-0.30-0.09-0.420.25
France-2.32-1.77-1.73-2.42-1.53-0.87
Germany-2.590.11-4.44-2.19-2.43-1.34
Greece1.790.620.17-0.29-0.25-0.20
Hong Kong SAR-1.750.01-2.30-4.08-1.23-0.99
Iceland-1.72-1.13-0.96-0.62-0.750.18
Ireland-0.310.84-0.36-0.93-0.69-0.68
Israel-1.46-0.98-1.43-1.57-0.42-0.26
Italy-1.97-4.05-3.14-2.30-1.38-1.32
Japan-0.311.40-2.63-0.62-0.17-0.47
Korea-1.19-1.74-1.98-2.32-0.370.69
Luxembourg1.140.701.110.93-0.080.69
Netherlands-0.14-0.05-1.26-0.36-0.320.63
New Zealand1.21-0.67-0.55-0.471.210.53
Norway-1.45-2.990.530.21-0.370.32
Portugal0.61-0.240.680.11-0.420.96
Singapore-0.75-0.69-3.08-2.67-0.330.48
Spain-0.79-1.020.190.14-0.54-0.14
Sweden-2.03-0.38-1.33-2.63-1.65-1.73
Switzerland0.542.38-0.69-0.020.280.54
Taiwan, Province of China-0.730.55-0.83-1.26-0.85-0.42
United Kingdom-1.57-0.67-0.21-0.82-1.76-0.42
United States-0.32-1.020.62-0.070.951.21
Inflation
Australia-0.31-0.270.32-0.06-0.090.23
Austria-0.35-0.280.060.86-0.28-0.32
Belgium1.131.841.340.710.930.25
Canada0.680.740.992.341.220.82
Cyprus-0.90-4.180.000.00-0.71-0.21
Denmark0.29-1.080.59-0.390.420.00
Finland2.262.170.771.811.60-0.37
France1.801.341.692.911.140.44
Germany1.071.710.651.421.11-0.12
Greece0.070.081.420.42-0.16-0.30
Hong Kong SAR-0.070.01-0.93-1.000.07-0.81
Iceland0.721.601.550.971.210.51
Ireland1.230.672.311.481.521.38
Israel1.770.411.530.401.63-1.23
Italy0.11-1.31-1.29-0.520.59-0.99
Japan1.54-1.09-0.13-0.522.64-0.06
Korea1.583.151.332.082.07-0.78
Luxembourg-0.190.26-0.280.15-0.06-0.55
Netherlands1.861.213.163.391.01-0.27
New Zealand1.140.220.530.190.81-0.54
Norway0.05-0.64-1.18-2.090.930.51
Portugal1.891.242.161.110.570.85
Singapore-0.37-0.74-0.570.08-0.201.30
Spain-0.12-0.45-0.30-0.180.70-0.12
Sweden0.64-0.251.300.340.470.54
Switzerland1.19-0.570.651.781.40-0.56
Taiwan, Province of China-2.19-1.45-0.92-0.53-1.47-0.09
United Kingdom1.240.15-0.40-0.212.27-0.09
United States-0.12-0.180.990.11-0.260.99
Current Account
Australia0.400.620.610.950.200.89
Austria0.34-1.370.77-0.270.780.47
Belgium-1.240.33-1.540.96-2.03-2.09
Canada2.600.851.070.952.942.26
Cyprus1.02-2.740.000.001.454.27
Denmark0.330.371.390.07-0.20-0.09
Finland0.070.14-0.750.94-1.25-1.45
France-0.57-1.40-1.79-5.10-0.170.10
Germany1.87-0.151.690.231.350.48
Greece-0.390.18-0.540.29-0.17-0.20
Hong Kong SAR-1.02-2.160.15-4.350.200.12
Iceland-0.610.44-0.98-0.58-1.85-1.83
Ireland0.530.870.780.31-0.29-0.10
Israel0.47-0.881.632.110.511.65
Italy2.541.280.790.390.22-0.48
Japan3.251.263.822.221.270.01
Korea-0.330.170.660.63-0.39-0.16
Luxembourg-0.33-0.24-0.800.31-1.79-1.73
Netherlands-1.17-1.75-3.00-2.490.080.11
New Zealand-1.69-0.56-1.310.171.07-1.25
Norway-0.19-1.46-0.53-0.890.010.27
Portugal-2.85-0.77-2.23-1.47-0.92-1.08
Singapore-5.52-1.17-1.44-0.67-1.95-3.66
Spain-1.57-1.78-1.84-2.67-0.51-0.59
Sweden1.912.781.931.940.26-0.47
Switzerland-0.311.49-1.060.54-3.47-3.16
Taiwan, Province of China-0.93-3.28-0.17-1.381.450.13
United Kingdom-2.10-0.91-0.73-0.83-1.27-0.57
United States0.09-0.18-0.150.400.870.00
Source: Author’s calculations.
Source: Author’s calculations.

There are only few cases where the WEO prediction of U.S. GDP growth appears to be correlated with the forecast errors. However, the ones that we find are of considerable interest. Indeed the evidence suggests that, for the Advanced Economies, 31 percent of the April current year forecasts and 24 percent of the September current year U.S. GDP forecasts generate a t-value above two and hence predict the forecast errors. This leads to a significantly positive t-statistic for 29 percent of the current-year forecast revisions in this region. In contrast there is no evidence that the U.S. GDP forecast has predictive power over the next-year forecast errors. The only other instance registering a greater than expected proportion of significant t-values is the current year forecasts for central and eastern Europe, where 33 percent of the t-values exceed a value of two. For many of the countries in this region, the revision to the current year forecast that takes place between April and September is predicted by the U.S. GDP forecast.

Turning to the WEO forecast of German output growth, interestingly this is positively correlated and significant in explaining forecast errors in a high proportion of countries in CIS and Mongolia (particularly for the next year forecast errors) but not to nearly the same extent in other regions.

With a few interesting exceptions—namely CIS and Mongolia for which predicted oil prices are positively correlated with forecast errors in GDP growth and Western Hemisphere and Advanced Economies for which a negative correlation emerges—the WEO forecasts of oil prices do not appear to be overly important in explaining forecast errors in output growth.

Interestingly, the global current account discrepancy is significant for 40 percent and 24 percent of the Advanced Economies in explaining the April current year and next year forecast errors, respectively.

There is evidence that the next year inflation forecast errors are linked to U.S. GDP forecasts, particularly for countries in central and eastern Europe, CIS, and Mongolia, Developing Asia, Western Hemisphere, and the Advanced Economies. Once again the WEO forecast of German output growth is significant in explaining the inflation forecast error for a very large proportion of the countries in the CIS and Mongolia region.

Both the WEO prediction of U.S. growth and oil prices affect the error in forecasting the current account for 20–30 percent of the countries in the Middle East. Furthermore, the global current account discrepancy affects the current account forecast errors in a greater than expected number of cases.

A. Output Gap

The output gap—measured as the difference between actual and potential GDP—plays an important role in the WEO forecasts. Implicit in these is an assumption that the output gap is eliminated after five years. If this assumption is unrealistic and leads to biased forecasts, then one would expect that the predicted value of the output gap itself be accountable for forecast errors. For example, if it takes longer to eliminate the output gap than assumed in the WEO, then the WEO will tend to overpredict forecasts for countries with large output gaps.

We have data on output gaps for the 29 Advanced Economies. For each of these we regress the forecast error on an intercept and the predicted output gap whose timing corresponds to the forecast with which it gets matched.

Table 7 presents the results in the form of t-statistics for current and next year forecast errors and forecast revisions. A pattern that stands out for the GDP forecasts is that the signs of the estimated t-values predominantly are negative. Around 15 percent of the t-statistics exceed two in absolute value. The large negative t-statistics for Germany, France, and Italy are particularly interesting since, as we shall see subsequently, these were also economies for which the WEO output growth forecasts were systematically upwards biased during the period. This finding suggests that the reduction in the output gap assumed in computing the WEO forecasts could lead to overpredictions: All three economies had large output gaps during the 1990s as did Japan—the output gap averaged -1.63, -1.99, -2.30, and -4.16 for France, Germany, Italy, and Japan, respectively. These were among the highest output gaps in the 29 countries. An assumption in the WEO forecasts that these output gaps would be reduced too fast might lead to a greater prediction of output growth and hence to an upward bias in the forecast.

Table 7.Efficiency Scores, by Country1/(Zero score best, three worst)
GDP GrowthInflationCurrent Account
Current-

Year
Next-

Year
Current-

Year
Next-

Year
Current-

Year
Next-

Year
Africa
Algeria131200
Benin001001
Botswana001001
Burkina Faso000000
Burundi211201
Cameroon011000
Cape Verde110011
Central African Republic210010
Chad000000
Comoros131002
Congo, Democratic Republic221011
Congo, Republic of121000
Cote d’Ivoire121001
Djibouti110001
Equatorial Guinea010011
Ethiopia100000
Gabon000000
Gambia, The001211
Ghana111111
Guinea110000
Guinea-Bissau001300
Kenya111311
Lesotho000111
Madagascar011100
Malawi112200
Mauritius101000
Morocco220011
Mozambique002210
Namibia111110
Niger001010
Nigeria001300
Rwanda001000
Sao Tome212310
Senegal101001
Seychelles100111
Sierra Leone221100
South Africa211000
Sudan022100
Swaziland001000
Tanzania122300
Togo221001
Tunisia000001
Uganda100010
Zambia102202
Zimbabwe110210
Regional average0.760.800.780.780.350.46
Central and Eastern Europe
Albania000010
Bulgaria210012
Croatia001000
Czech Republic011112
Estonia002200
Hungary110200
Latvia102201
Lithuania112200
Macedonia211000
Malta010001
Poland112200
Romania221301
Slovak Republic210011
Slovenia113211
Turkey000010
Regional average0.870.7311.070.400.60
CIS and Mongolia
Armenia111211
Azerbaijan121210
Belarus122200
Georgia111200
Kazakhstan122201
Kyrgyz Republic012111
Moldova212200
Mongolia021301
Russia002311
Tajikistan122321
Ukraine012212
Uzbekistan222201
Regional average0.831.421.672.170.580.75
Developing Asia
Bangladesh000000
China131100
India101212
Indonesia000021
Malaysia001100
Pakistan110212
Papua New Guinea001201
Philippines111100
Sri Lanka000110
Thailand000010
Vanuatu110100
Vietnam011200
Regional average0.420.580.501.080.500.50
Middle East
Bahrain120301
Egypt112101
Iran101100
Jordan100032
Kuwait101102
Lebanon022310
Oman100100
Qatar001100
Saudi Arabia001202
Syrian Arab Republic011001
United Arab Emirates000000
Regional average0.550.550.821.180.360.82
Western Hemisphere
Argentina002001
Bolivia131011
Brazil102222
Chile000000
Colombia222202
Costa Rica000102
Dominican Republic110000
Ecuador100300
El Salvador021200
Guatemala213301
Mexico013210
Panama001100
Peru102200
Trinidad and Tobago001300
Uruguay002311
Venezuela000311
Regional average0.560.631.251.690.380.69
Advanced Economies
Australia101210
Austria001011
Belgium121001
Canada000111
Cyprus100003
Denmark101111
Finland122111
France110012
Germany120001
Greece112200
Hong Kong SAR001200
Iceland200001
Ireland120102
Israel111010
Italy120001
Japan010000
Korea000000
Netherlands120002
New Zealand110111
Norway000100
Portugal220002
Singapore110112
Spain011111
Sweden020112
Switzerland112122
Taiwan, Province of China001203
United Kingdom110001
United States000010
Regional average0.710.890.500.640.501.11

Aggregate score of the indicator variables for the three efficiency tests—tests for bias, serial correlation, and predictability by means of any of the four predictor variables examined earlier (Table 5). An indicator value of 1 indicates that the test was failed (rejection of the null hypothesis of an efficient forecast).

Aggregate score of the indicator variables for the three efficiency tests—tests for bias, serial correlation, and predictability by means of any of the four predictor variables examined earlier (Table 5). An indicator value of 1 indicates that the test was failed (rejection of the null hypothesis of an efficient forecast).

The sign of the regression coefficient of the output gap is predominantly positive in the case of the inflation forecast errors, i.e. the opposite sign of what was found for the GDP forecasts. Hence the WEO tends to underpredict inflation, the larger the output gap, i.e. the greater an economy’s unused capacity. This effect can be quite large and is borderline significant for countries such as France, Germany, and Korea.

Finally, turning to the regression results for the current account, there are many instances with large and significant predictability from the output gap over subsequent forecast errors, although the sign of the regression coefficient varies quite a bit. Countries for which a significant degree of predictability is found include Hong Kong, Japan, the Netherlands, Singapore, and Sweden.

B. Overall Scores, by Country and Region

So far we have tested the efficiency properties of the growth, inflation and current account forecasts and reported results aggregated by region. It is also of interest to study forecast efficiency at the level of the individual countries. Indications of problems for a particular country model may suggest that the model should be modified. For this purpose, we create an aggregate score. Each of three efficiency tests—tests for bias, serial correlation, and predictability by means of any of the four predictor variables examined—is associated with an indicator variable that is one if the relevant t-statistic exceeds two in absolute value. Otherwise the indicator variable equals zero. This means that each country gets a score of zero (no detectable problem) and three (detectable problems along all dimensions). We produce these scores for current year and next year forecasts of GDP growth, inflation, and the current account.

Results for individual countries along with regional averages are shown in Table 8. Interestingly, it is not the case that the Advanced Economies is the region in which the average score is lowest. This may have to do with the greater importance of measurement errors and outliers in some of the other regions. It can be easier to detect relatively small inefficiencies in economies not affected by outliers.

Table 8.Sign Tests for Directional Accuracy(Average across regions except for last column)
Fraction of Observations

with Correct Sign
Benchmark

Estimates
PT Statistic1/Fraction of Significant

PT Test Statistics2/
Real GDP
April current-year forecast errors
Africa0.730.521.810.46
Central and Eastern Europe0.800.532.130.67
CIS and Mongolia0.670.521.160.23
Developing Asia0.780.522.120.65
Middle East0.770.512.060.62
Western Hemisphere0.710.511.610.38
Advanced economies0.790.502.280.66
September next-year forecast errors
Africa0.660.501.400.24
Central and Eastern Europe0.700.531.260.36
CIS and Mongolia0.610.550.550.08
Developing Asia0.680.511.350.26
Middle East0.710.511.550.29
Western Hemisphere0.650.501.260.18
Advanced economies0.620.481.190.14
Export Volume
April current-year forecast errors
Africa0.750.511.880.57
Central and Eastern Europe0.760.521.910.36
CIS and Mongolia0.810.512.100.45
Developing Asia0.740.501.880.63
Middle East0.850.502.610.79
Western Hemisphere0.790.502.260.71
Advanced economies0.760.492.110.61
September next-year forecast errors
Africa0.700.491.620.40
Central and Eastern Europe0.730.501.730.50
CIS and Mongolia0.750.511.660.46
Developing Asia0.740.511.830.58
Middle East0.710.501.570.29
Western Hemisphere0.720.501.710.45
Advanced economies0.690.501.540.36
Import Volume
April current-year forecast errors
Africa0.740.511.840.47
Central and Eastern Europe0.790.502.150.73
CIS and Mongolia0.720.501.530.36
Developing Asia0.740.501.900.53
Middle East0.810.512.290.71
Western Hemisphere0.810.512.370.77
Advanced economies0.770.502.160.64
September next-year forecast errors
Africa0.730.501.830.46
Central and Eastern Europe0.780.512.020.60
CIS and Mongolia0.730.511.560.55
Developing Asia0.740.511.830.41
Middle East0.770.521.990.43
Western Hemisphere0.770.512.070.64
Advanced economies0.700.491.650.46
Inflation
April current-year forecast errors
Africa0.740.521.830.44
Central and Eastern Europe0.820.622.080.73
CIS and Mongolia0.780.621.650.54
Developing Asia0.690.511.470.45
Middle East0.680.501.370.23
Western Hemisphere0.730.531.760.34
Advanced economies0.760.531.890.55
September next-year forecast errors
Africa0.690.521.460.24
Central and Eastern Europe0.700.590.920.21
CIS and Mongolia0.790.611.650.36
Developing Asia0.630.511.080.21
Middle East0.620.491.060.14
Western Hemisphere0.680.531.440.40
Advanced economies0.670.511.360.24
Current Account (in percent of GDP)
April current-year forecast errors
Africa0.600.510.700.11
Central and Eastern Europe0.710.501.540.29
CIS and Mongolia0.670.511.170.08
Developing Asia0.700.501.620.44
Middle East0.690.531.290.45
Western Hemisphere0.650.501.190.20
Advanced economies0.630.510.930.14
September next-year forecast errors
Africa0.590.500.730.13
Central and Eastern Europe0.640.491.090.20
CIS and Mongolia0.620.491.020.17
Developing Asia0.650.501.160.30
Middle East0.660.491.300.14
Western Hemisphere0.600.500.750.13
Advanced economies0.540.510.240.07
Source: Author’s calculations.

The PT test is a sign that proposed by Pesaran and Timmermann (1992).

For all countries included in a region.

Source: Author’s calculations.

The PT test is a sign that proposed by Pesaran and Timmermann (1992).

For all countries included in a region.

With the exception of the Advanced Economies, the inflation forecasts generate the highest “problem scores.” These are on average above one in most regions, suggesting that at least one of the efficiency tests is rejected. Furthermore, for most regions the next-year forecasts generate higher average scores than the current-year forecasts, so the greatest inefficiencies are to be found in the next year forecasts.

VII. Directional Accuracy

The anticipation of turning points is acknowledged to be one of the most difficult exercises in economic forecasting. If done with any measure of success it is also one of the potentially most rewarding ones. Most obviously, the ability to foresee a recession will allow policy makers to take actions and install remedies that will cushion the downturn in economic activity.

Defining turning points is made difficult by the fact that official definitions of recession periods (such as the NBER recession indicator available for the United States) are not widely published for the majority of countries included in our sample. Furthermore, recession periods do not always coincide with calendar years and our forecasts are only available on an annual basis. To deal with these problems we simply study the extent to which the WEO forecasts predicted the sign of the variables of interest. Although this measure is subject to the criticism that it need not accurately track recessions, it has the advantage that it is more robust to the type of outliers that are so prevalent in many of the regions outside the Advanced Economies.

The test statistic underlying our analysis is the sign test proposed by Pesaran and Timmermann (1992) which takes the form:

where P^ is a sample estimate of the proportion of positive signs that are correctly predicted, P^* is an estimate of the probability of correctly predicting the signs under the assumption that predictions and realizations are independently distributed, and V^(P^), V^(P^*) are consistent estimates of the variances of P^ and P^*, respectively. P^* sets a natural benchmark for sign predictions for a forecaster with no genuine prediction skills. It can be consistently estimated by P^*=P^YP^Y^+(1P^Y(1P^y¯), where P^y is the estimated proportion of positive values of the target (outcome) variable while P^y¯ is the estimated proportion of positive values of the predictions.

For example, if the proportion of positive realizations is 80 percent, then a forecaster without any skill that predicts positive growth, say, 90 percent of the time by flipping a biased coin can expect to correctly predict 74 percent of the signs.8 The test statistic corrects for this by benchmarking forecasts relative to P^*.

Results from applying this test are shown in Table 9. The table shows the outcomes averaged within each of the seven regions and categorized by the time of the prediction. The first column shows the percentage of correctly predicted signs followed by a column of benchmark estimates, P^*. If the forecasts contain predictive sign information we should expect the first column to exceed the second column and the higher the differential, the more precise the sign information embedded in the forecast. Columns three and four show the average of the PT test statistic in addition to the percentage of PT statistics across each region that exceeds two.

Table 9.Real GDP: Significance of Forecast Revisions After Executive Board Meeting(Average across regions except for fractions)
Fractions of
T-Valuesβ^
tβ^<2tβ^>2Coefficients>0R-SquareMSE-Ratio
April current-year forecast errors
Africa0.000.130.810.110.83
Central and Eastern Europe0.000.070.600.090.85
CIS and Mongolia0.000.150.540.070.48
Developing Asia0.000.150.500.160.72
Middle East0.000.000.620.030.77
Western Hemisphere0.000.130.530.080.64
Advanced economies0.000.520.900.230.81
September current-year forecast errors
Africa0.000.070.410.070.55
Central and Eastern Europe0.000.230.690.140.69
CIS and Mongolia0.000.330.670.180.67
Developing Asia0.000.270.550.150.43
Middle East0.000.600.800.360.75
Western Hemisphere0.000.380.670.240.74
Advanced economies0.000.410.830.270.72
April next-year forecast errors
Africa0.000.100.630.080.92
Central and Eastern Europe0.130.000.270.140.77
CIS and Mongolia0.000.000.380.040.36
Developing Asia0.000.040.480.070.63
Middle East0.070.000.430.071.64
Western Hemisphere0.030.120.520.080.66
Advanced economies0.030.140.620.120.95
September next-year forecast errors
Africa0.040.040.380.060.81
Central and Eastern Europe0.000.070.600.080.63
CIS and Mongolia0.000.000.380.010.55
Developing Asia0.040.170.480.150.74
Middle East0.000.140.570.090.73
Western Hemisphere0.030.060.450.090.80
Advanced economies0.000.070.590.070.90
Source: Author’s calculations
Source: Author’s calculations

First consider the directional accuracy of the GDP forecasts. Moving from April next year forecasts through the September current year forecasts, as the information underlying the WEO forecasts becomes more accurate one should expect the percentage of correctly predicted forecasts to improve and the average PT sign test to go up. While this is generally found to be true, it is also clear that by far the largest improvement takes place in the revisions when moving from the September next year forecast to the April current-year forecast or from the April current-year forecast to the September current-year forecast. Conversely, there is little evidence that next-year forecast revisions occurring between April and September result in an improved ability to forecast the sign of future GDP growth.

The April and September next-year GDP growth forecasts do reveal some ability to forecast the direction of future economic activity. The benchmark percentage of correctly predicted signs (which embodies no information other than knowledge of the average, long-term percentage of years with positive growth) is around 50 percent while the proportion of predictions with correct signs lies between 62 and 71 percent. This gives rise to PT test statistics that are significant for about 20 percent of the countries.

Directional accuracy improves vastly when moving to the same-year forecasts. The proportion of correctly predicted signs now varies between 68 and 78 percent for the April forecasts and between 76 and 90 percent for the September forecasts. Furthermore, the average values of the PT test statistic and the percentage of these that are significant also go up by a large amount. For example, for the Advanced Economies the proportion of correctly predicted signs increases from 62 percent (next year forecasts) to 78 percent (April current-year forecasts) and 90 percent (September current-year forecasts). The “hit ratio”—i.e. the percentage of correctly predicted signs—obtains a high level of significance for all regions with exception of CIS and Mongolia.

Turning to the inflation forecasts, a similar pattern is observed: little improvement in the directional accuracy of the forecasts when moving from the April to the September next-year forecasts, followed by more significant improvements in the current-year forecasts. Interestingly, while the hit rate of the Advanced Economies’ next-year inflation forecasts is higher than was observed for the next-year GDP forecasts, the improvement when moving to the April current-year forecasts is smaller, so the directional accuracy of the September same-year forecasts is basically identical for the two sets of forecasts.

Finally, the directional accuracy of the current account forecasts is generally lower than that observed for inflation or output growth. As expected this gives rise to smaller average PT test statistics and a lower percentage of these that are significant at the 5 percent level.

VIII. Revisions from Board to Published Forecasts

WEO forecasts are published twice a year in April and September. Several rounds of forecast revisions precede the published version. A first set of predictions is presented to the IMF Board in February and July each year, preceding the April and September WEO publication. To assess the informational value of forecast revisions that occur between the Board version and the published version, we obtained data on Board forecasts of current-year GDP growth in February and next-year Board forecasts of GDP growth reported in July. We refer to these forecasts as y^t,tFeb and y^t+1,tJuly, respectively. Further, let the forecast revisions from the Board to the published WEO forecasts be given by revt,tpubBoard and revt+1,tpubBoard. If the revisions occurring between the Board and published forecasts contain useful information, we should expect that they help predict the errors in the original Board forecasts, defined as et,tBoard=yty^t,tFeb and et+1,tBoard=yt+1y^t+1,tJuly. We test this proposition through the regressions

If the revisions incorporated in the published WEO forecasts do not add any value to the original Board forecast then we should expect to find β−coefficients near zero. Conversely, we would expect to find significant and positive values of β and non-zero R2-values in case the revisions contain valuable information. Estimation results based on (10) are reported in Table 10. The current-year forecast errors for the Advanced Economies reveal strong evidence that the Board-to-Publication revision contains valuable information that not only is 20 significantly correlated with the forecast error for around 50 percent of the countries but has the required positive sign for between 80 and 90 percent of the countries. The large R2-value of around 0.25 is further testimony to this effect and suggests that 25 percent of the current-year February or July forecast error can be explained by the revision between the Board and published version.

Table 10.Description Statistics for Forecast Errors During 2001–2003 Recession and Recovery(Average across countries in region)
Forecast Error
Previous-YearCurrent-Year
OutcomeAprilSeptemberAprilSeptember
Real GDP
2001 recession results
Africa4.33-0.58-0.37-0.76-0.66
Central and Eastern Europe2.68-2.08-2.23-1.40-0.92
CIS and Mongolia7.873.933.312.922.74
Developing Asia2.63-2.08-2.19-1.52-1.03
Middle East3.520.420.00-0.34-0.61
Western Hemisphere1.14-2.97-2.90-2.62-1.98
Advanced economies1.53-2.05-2.17-1.32-0.35
2002 recovery results
Africa3.10-1.64-1.93-1.55-1.16
Central and Eastern Europe4.34-0.40-0.470.350.56
CIS and Mongolia6.251.431.751.551.06
Developing Asia3.36-1.32-1.22-0.47-0.38
Middle East2.39-1.38-1.580.210.03
Western Hemisphere0.14-3.88-3.64-0.310.26
Advanced economies1.89-1.44-0.92-0.010.11
2003 recovery results
Africa4.12-0.55-0.66-0.130.25
Central and Eastern Europe4.29-0.36-0.220.160.20
CIS and Mongolia8.914.264.003.583.43
Developing Asia4.980.240.330.510.50
Middle East5.261.471.521.151.23
Western Hemisphere2.75-0.96-0.011.181.62
Advanced economies1.76-1.59-1.11-0.160.40
Export Volume
2001 recession results
Africa6.38-3.31-2.43-2.01-1.60
Central and Eastern Europe5.46-3.94-2.41-2.89-2.48
CIS and Mongolia2.11-2.101.912.54-10.58
Developing Asia1.31-6.03-5.86-5.40-2.98
Middle East2.520.45-0.48-3.20-4.42
Western Hemisphere0.33-6.45-6.41-5.83-5.12
Advanced economies0.16-6.54-7.21-5.70-2.48
2002 recovery results
Africa4.89-1.1022.5823.02-2.53
Central and Eastern Europe6.11-1.14-1.43-0.350.09
CIS and Mongolia6.225.24-6.611.831.72
Developing Asia6.78-0.60-0.241.821.70
Middle East5.562.561.822.502.58
Western Hemisphere-1.81-8.57-9.22-7.58-4.87
Advanced economies2.60-3.48-2.550.120.88
2003 recovery results
Africa5.2823.290.50-1.38-1.65
Central and Eastern Europe12.845.055.356.185.92
CIS and Mongolia10.173.302.894.232.84
Developing Asia10.904.034.956.946.10
Middle East6.593.853.28-2.163.29
Western Hemisphere3.32-2.72-1.993.343.12
Advanced economies3.29-2.70-2.43-0.970.17
Import Volume
2001 recession results
Africa6.362.211.56-3.6922.10
Central and Eastern Europe1.82-6.06-5.33-5.67-2.53
CIS and Mongolia13.139.199.019.182.56
Developing Asia-0.17-6.57-8.71-7.52-5.79
Middle East7.201.895.364.445.44
Western Hemisphere2.10-3.47-3.27-4.73-3.51
Advanced economies-1.28-7.78-8.77-7.23-3.49
2002 recovery results
Africa4.23-0.3723.3620.95-6.15
Central and Eastern Europe8.622.091.631.712.30
CIS and Mongolia6.634.48-0.71-3.66-4.17
Developing Asia5.86-0.91-0.26-0.170.65
Middle East11.898.397.915.784.62
Western Hemisphere-7.20-12.10-12.47-9.43-4.45
Advanced economies2.17-4.08-2.63-0.580.12
2003 recovery results
Africa4.7325.460.17-2.52-4.06
Central and Eastern Europe13.196.196.337.807.30
CIS and Mongolia15.039.268.579.419.56
Developing Asia10.351.903.414.334.06
Middle East2.08-2.28-2.75-1.771.97
Western Hemisphere4.88-0.12-0.264.082.56
Advanced economies4.50-1.53-1.24-0.120.85
Inflation
2001 recession results
Africa18.4212.589.317.412.36
Central and Eastern Europe10.494.884.230.760.91
CIS and Mongolia16.43-8.29-1.56-2.83-2.03
Developing Asia4.911.080.430.010.10
Middle East1.39-2.81-3.20-2.24-1.85
Western Hemisphere7.011.52-2.370.31-1.01
Advanced economies2.530.430.100.11-0.05
2002 recovery results
Africa11.005.293.191.320.22
Central and Eastern Europe7.401.090.24-0.73-0.72
CIS and Mongolia11.141.180.830.870.19
Developing Asia7.172.452.342.132.22
Middle East2.34-2.97-3.05-2.32-2.72
Western Hemisphere6.731.671.690.660.26
Advanced economies2.170.030.050.220.01
2003 recovery results
Africa18.1812.340.271.030.21
Central and Eastern Europe5.12-1.35-1.61-0.97-0.49
CIS and Mongolia9.080.810.62-0.04-0.81
Developing Asia6.681.761.640.41-1.10
Middle East3.32-0.20-0.59-0.43-0.84
Western Hemisphere9.753.380.680.61-0.12
Advanced economies1.81-0.02-0.06-0.26-0.05
Current Account (in Percent of GDP)
2001 recession results
Africa-7.79-0.17-0.290.210.06
Central and Eastern Europe-5.01-0.250.04-0.050.67
CIS and Mongolia-2.895.514.300.75-0.81
Developing Asia0.422.352.442.141.86
Middle East6.836.682.400.56-0.84
Western Hemisphere-7.67-3.72-2.52-1.73-0.05
Advanced economies1.670.380.330.890.50
2002 recovery results
Africa-6.800.080.960.731.32
Central and Eastern Europe-5.55-0.87-0.16-0.33-0.19
CIS and Mongolia-2.680.760.102.171.38
Developing Asia-0.112.582.252.962.79
Middle East3.231.57-0.791.680.40
Western Hemisphere-6.62-1.021.001.840.73
Advanced economies2.421.711.210.930.76
2003 recovery results
Africa-5.022.382.362.641.89
Central and Eastern Europe-5.78-0.81-0.78-0.50-0.24
CIS and Mongolia-2.842.662.121.991.34
Developing Asia1.093.893.452.351.91
Middle East6.905.783.23-1.050.88
Western Hemisphere-4.601.930.790.92-0.04
Advanced economies2.450.840.660.25-0.07
Source: Author’s calculations.
Source: Author’s calculations.

Much lower levels of significance are obtained for the next-year forecasts where, in the case of the Advanced Economies, close to 60 percent of the β−estimates are positive and only 10 percent of the coefficients exceed two. Furthermore, the average R2−value now declines to a level near 0.10.

The R2 values do not in themselves quantify the degree of improvement in the WEO forecast from the Board to the published version. A better measure of this is the ratio of MSE-values based on t=1T(et,tApr)2/t=1T(et,tBoard)2 and t=1T(et+1,tSep)2/t=1T(et+1,tBoard)2, respectively, where T is the sample size. The final column in Table 10 show these ratios. Values below unity indicate that the WEO forecast gets more precise from the Board to the published version and the extent to which the ratio is below unity is a measure of the improvement. For the February/April same-year forecasts the average ratio is around 0.80 for the Advanced Economies. This declines to a value near 0.70 for the July/September same-year forecasts, but is closer to 0.95 for the next-year forecasts. These values suggest that much valuable information is learned about current-year economic growth between the time when the Board forecast is reported and the time of the official publication. Far less information is learnt about next-year economic growth between these dates as witnessed by the R2−values near 0.95.

Turning to the countries outside the Advanced Economies region, in general the percentage of positive coefficients in the Board-to-Publication revision regressions in (10) is somewhat lower as is the fraction of estimates that is statistically significant. In fact, only about 10–15 percent and 5–10 percent of the current-year and next year coefficient estimates generate positive t-values that exceed two. Interestingly, the MSE ratio tends to be somewhat lower than was found for the Advanced Economies, especially for the next-year forecasts, suggesting a significant improvement in the next year forecasts between the Board and the published forecasts for the other regions.

IX. Recent Performance of WEO Forecasts

Table 11 presents results for the recent performance of the WEO forecasts during the 2001 recession and 2002, 2003 recovery years. The recession and recovery year labels should be loosely interpreted as they are not uniformly accurate descriptors across the various regions. For example, although U.S. GDP growth was only 0.25 percent in 2001, this year saw average growth close to 8 percent in CIS and Mongolia, 4.3 percent in Africa and 3.5 percent in the Middle East—all examples of robust growth.

Table 11.Forecasts for G-7 Economies: Descriptive Statistics for Long Sample
VariableMeanForecast Errors
MeanStandard

Deviation
Absolute

Forecast Error
Forecast

Error
Standard

Deviation
Serial

Correlation1/
Real GDP
April current-year forecast analysis
Canada2.862.240.92-0.181.21-0.19
France2.341.790.64-0.131.000.11
Germany1.971.920.90-0.291.220.17
Italy1.941.990.91-0.211.20-0.07
Japan3.322.991.21-0.061.53-0.05
United Kingdom1.761.970.79-0.111.030.03
United States2.742.270.760.050.90-0.10
September next-year forecast analysis
Canada2.672.181.55-0.571.88-0.07
France2.181.551.07-0.551.50-0.09
Germany1.911.951.51-0.821.92-0.10
Italy1.922.021.44-0.511.870.13
Japan3.002.801.67-0.802.570.13
United Kingdom1.901.971.20-0.341.580.41
United States2.652.291.320.031.80-0.01
Inflation
April current-year forecast analysis
Canada4.673.671.050.121.330.28
France5.253.950.690.160.980.03
Germany3.292.110.59-0.080.75-0.07
Italy8.955.950.900.511.450.15
Japan2.504.471.04-0.451.750.16
United Kingdom7.355.821.170.291.64-0.41
United States4.202.600.38-0.050.550.10
September next-year forecast analysis
Canada4.703.881.490.182.020.41
France5.414.461.000.291.390.52
Germany3.091.950.64-0.120.780.26
Italy9.026.041.770.772.450.37
Japan2.354.551.70-0.502.81-0.05
United Kingdom7.235.941.420.692.230.38
United States4.302.710.84-0.111.220.38
Current Account
April current-year forecast analysis
Canada-4.6410.524.050.495.650.07
France7.1315.584.430.126.440.37
Germany6.2525.419.570.0013.720.28
Italy1.9714.836.64-1.119.18-0.14
Japan61.1449.1212.630.2616.54-0.05
United Kingdom-8.8513.915.130.827.350.03
United States-135.21154.9515.70-1.2818.97-0.15
September next-year forecast analysis
Canada-4.1611.334.661.086.870.52
France6.8114.746.76-0.488.770.51
Germany8.4224.7013.22-0.1719.230.42
Italy1.0415.0810.47-2.1614.340.33
Japan60.6649.6817.66-0.0922.690.29
United Kingdom-6.5613.626.272.238.670.37
United States-129.96153.1426.41-5.1337.840.15
Import Volume
April current-year forecast analysis
Canada6.277.354.851.695.95-0.05
France4.705.963.29-0.274.36-0.18
Germany4.955.633.75-0.875.33-0.09
Italy4.346.833.97-0.604.99-0.12
Japan5.358.043.64-0.424.39-0.18
United Kingdom4.985.062.970.453.53-0.07
United States7.037.833.961.374.96-0.07
September next-year forecast analysis
Canada5.856.905.451.706.620.12
France4.935.533.99-0.185.10-0.09
Germany5.065.083.78-0.234.560.08
Italy4.126.944.65-1.206.02-0.05
Japan4.867.795.25-0.746.590.09
United Kingdom4.835.073.510.244.200.12
United States6.997.874.881.615.80-0.06
Export Volume
April current-year forecast analysis
Canada5.466.193.381.054.700.05
France4.964.773.11-0.144.02-0.47
Germany5.005.793.54-0.444.76-0.18
Italy4.594.823.75-0.694.46-0.12
Japan5.206.544.190.855.16-0.22
United Kingdom3.973.842.79-0.173.54-0.44
United States5.757.592.480.243.45-0.38
September next-year forecast analysis
Canada6.125.654.221.585.440.18
France4.804.553.25-0.624.29-0.21
Germany5.185.373.680.054.82-0.21
Italy4.494.764.62-0.905.280.13
Japan5.326.655.110.806.07-0.29
United Kingdom4.133.702.97-0.293.460.00
United States5.837.703.510.544.670.01
Source: Author’s calculations.

First-order serial correlation coefficient.

Source: Author’s calculations.

First-order serial correlation coefficient.

Column one in Table 11 shows the mean value of the actual values computed across countries in the various regions. Columns two through five show the 21 average forecast error computed in April and September of the previous year (labeled (-1)) and April and September of the current year (labeled (0)).

First consider the GDP forecasts. The WEO forecasts overpredicted growth in 2001 in all regions with the exception of CIS and Mongolia and the September 2000 forecast of 2001 growth in the Middle East. There are clear patterns in the forecast revisions: predictions of 2001 GDP growth were not revised downwards to any noticeable degree between April and September 2000. However, there was a substantial reduction in the mean growth forecast between September 2000 and April 2001 and again from April to September 2001. For example, the mean forecast error of the September 2000 forecast of 2001 growth in the Advanced Economies was 2.17 percent. This gets reduced to 1.32 percent and 0.35 percent in the April and September 2001 forecasts, respectively.

Results for the recovery in 2002 bear some resemblance to the 2001 recession. The April and September previous-year (2001) WEO forecasts overpredicted growth in six of the seven regions, followed by significant adjustments in the April 2002 current-year forecast which gets the forecast right to within 0.4 percent of the actual value in four regions. One clear difference to the results for 2001 is that there is now evidence of small underpredictions in the September current year forecast for five of the seven regions, the only exceptions being Africa and Developing Asia for which the WEO forecasts on average overpredicted GDP growth.

The 2003 forecasts were very precise for Africa, central, and eastern Europe and developing Asia where all forecast errors lie within 0.7 percent of the realization. Once again the familiar pattern of previous-year April and September overpredictions followed by significant reductions that bring the prediction close to the actual value in the April and September same-year value is observed for the Advanced Economies.

Turning to the inflation forecasts, forecast errors are, unsurprisingly, more volatile in many of the regions than the corresponding GDP forecast errors. Nevertheless, some patterns emerge from the data. For Africa the previous year April forecast underpredicted the actual value in all three years by a wide margin ranging from 5.3 percent in 2002 to 12.6 percent in 2001. However, this forecast error comes down significantly between April and September of the previous year and gets further reduced in the subsequent same-year forecasts. Inflation forecasts were generally quite accurate for 2003. Moreover, the WEO inflation forecasts for the Advanced Economies are very accurate in all years. They fall within 0.45 percent of the actual value and are typically much closer than this to the outcome.

Finally, the WEO forecasts of the current account (as a percentage of GDP) show a general tendency to underpredict during the 2001–2003 period for countries in CIS and Mongolia, Developing Asia, Middle East, and the Advanced Economies. In general the numbers are not that large and the absolute value of the forecast error tends to get reduced as the point of the prediction draws closer to the time of the outcome.

X. Long-Run Forecasting Performance for G-7 Economies

Artis (1997) undertook a postmortem analysis of the WEO forecasting performance for the G-7 economies using data from the early seventies up to 1994. The decade that has passed since this analysis has produced new data that facilitates an analysis of the WEO forecasting performance around subsequent events.

As in the earlier analysis, we have data on five variables, namely GDP growth, inflation, export and import volume and the current account balance.9 In the case of the GDP growth and inflation data the current year forecasts date back to 1971 while next-year forecasts date back to 1973. In the case of the import and export volume and the current account balance, the current year forecasts only date back to 1972 while next-year forecast begin in 1973. Data on outcomes extend up to 2003 so we have between 33 and 31 annual observations. Not a very long sample by any common yardstick, but much longer than the WEO data and Artis’ original data.10

The WEO reports current year and next year forecasts in both April and September each year so a decision has to be made on which of these to use in the analysis. We follow Artis (1997) and focus on the current-year forecast published in April and the next-year forecast published in September.

A. GDP Growth

First consider the full-sample results for the GDP growth forecasts reported in Table 12. Current year mean absolute forecast errors range from 0.64 for France to 1.21 for Japan while the RMSFE-values vary from 0.90 (USA) to 1.53 (Japan). Forecast errors for all countries except the US have a slightly negative mean, representing over-predictions.

Table 12.G-7 Countries: Forecasting Performance Relative to Simple Benchmarks(Ratio of WEO RMSFE over the RMSFE-values produced by naïve random walk and recursive mean forecasts)
Random WalkRecursive Mean
Full SampleSubperiod 1Subperiod 2Full SampleSubperiod 1Subperiod 2
Real GDP
April current-year forecasts
Canada0.440.410.540.520.500.58
France0.470.440.570.520.500.62
Germany0.480.450.560.600.570.67
Italy0.410.360.660.520.480.70
Japan0.470.400.650.440.460.42
United Kingdom0.480.460.540.490.490.49
United States0.310.280.450.370.330.50
September next-year forecasts
Canada0.680.610.860.810.750.93
France0.780.671.030.870.741.17
Germany0.760.670.970.880.771.17
Italy0.690.641.010.820.771.06
Japan0.870.830.990.800.880.66
United Kingdom0.780.730.910.720.640.92
United States0.610.560.790.690.620.96
Inflation
April current-year forecasts
Canada0.670.630.790.350.450.26
France0.740.711.020.240.440.10
Germany0.540.490.650.330.390.27
Italy0.590.590.570.240.370.10
Japan0.540.530.930.370.430.16
United Kingdom0.360.340.740.250.300.13
United States0.360.350.480.200.270.11
September next-year forecasts
Canada0.930.930.950.490.660.31
France0.850.811.360.290.470.13
Germany0.610.460.840.390.350.41
Italy1.071.081.000.390.640.16
Japan0.850.831.280.520.590.25
United Kingdom0.520.501.070.350.420.17
United States0.780.750.980.410.530.23
Current Account
April current-year forecasts
Canada0.960.990.960.530.880.51
France0.940.641.050.410.710.39
Germany0.830.650.850.530.320.59
Italy0.790.500.820.610.560.61
Japan0.780.680.800.330.290.35
United Kingdom0.780.760.800.520.670.47
United States0.440.610.410.120.220.11
September next-year forecasts
Canada1.130.921.150.600.750.60
France1.200.991.260.591.120.55
Germany1.141.091.150.770.580.81
Italy1.370.981.430.941.070.93
Japan1.080.851.130.450.340.48
United Kingdom0.940.960.930.630.750.58
United States0.830.810.830.250.280.24
Import Volume
April current-year forecasts
Canada0.630.530.860.760.641.02
France0.510.250.770.670.331.01
Germany0.770.540.860.910.611.04
Italy0.510.440.610.660.540.92
Japan0.420.350.610.490.430.66
United Kingdom0.570.440.850.650.520.88
United States0.470.360.790.570.440.91
September next-year forecasts
Canada0.730.631.000.910.791.25
France0.640.540.810.840.721.05
Germany0.720.680.740.840.770.89
Italy0.600.560.670.780.681.02
Japan0.660.590.850.720.621.06
United Kingdom0.680.660.740.740.710.87
United States0.550.450.940.660.551.05
Export Volume
April current-year forecasts
Canada0.610.520.800.710.620.91
France0.610.450.730.760.501.03
Germany0.570.350.830.750.451.08
Italy0.670.620.730.840.780.92
Japan0.540.480.630.750.640.97
United Kingdom0.650.450.910.790.531.14
United States0.430.370.530.430.360.53
September next-year forecasts
Canada0.820.691.060.900.821.02
France0.710.630.800.890.781.01
Germany0.630.550.900.770.680.98
Italy0.820.730.931.040.981.10
Japan0.610.620.610.850.820.92
United Kingdom0.730.640.880.840.721.09
United States0.560.440.740.520.370.96
1/ Source: Author’s calculations.
1/ Source: Author’s calculations.

Biases in the next-year forecast error are again negative for six of the seven economies. Next year biases are also greater than the biases observed for the current-year forecasts. There is little evidence of serial correlation in the full sample forecast errors, the only exception being the next-year forecasts for the United Kingdom which generate a first-order serial correlation coefficient of 0.41.

We measure forecasting performance relative to two naive benchmarks by computing the ratio of RMSFE-values of the WEO forecasts relative to a random walk forecast that uses the most recent (previous year) observation and relative to the recursive sample average of the target variable computed using the most recently available data (again the outcome during the previous year). The results, reported in Table 13, show that the WEO forecasts generate RMSFE-values around 40 percent the size of the random walk forecast or around 50 percent of the size of the RMSFE-value associated with the recursive mean forecast. Compared to the current year ratios, the next year RMSFE ratios increase by between 0.10 and 0.20. This means that the next year RMSFE ratios are close to 0.60 relative to the random walk forecasts while they fall between 0.60 and 0.80 for the recursive mean forecasts.

Table 13.G-7 Countries: Predictive Efficiency Test Results(t-statistics)
April Current-Year ForecastSeptember Next-year Forecast
Bias 1/Serial

Correlation 2/
Lagged

Realization 3/
Bias 1/Serial

Correlation 2/
Lagged

Realization 3/
Real GDP
Canada-0.84-1.38-1.12-1.76-0.32-0.15
France-0.761.03-1.22-2.20-0.42-1.75
Germany-1.410.99-0.40-2.62-0.45-0.86
Italy-1.02-0.41-0.89-1.590.46-0.28
Japan-0.24-0.23-1.30-1.821.49-0.77
United Kingdom-0.630.141.41-1.221.930.60
United States0.30-0.45-2.240.09-0.07-1.45
Post-1990 Sub Sample
Canada-0.79-0.63-0.02-1.540.70-0.03
France-1.24-0.621.53-2.25-1.55-1.83
Germany-0.530.111.36-2.34-1.83-2.23
Italy-2.04-0.81-0.95-4.40-0.92-0.80
Japan0.03-0.30-0.13-2.01-0.83-1.29
United Kingdom-0.120.85-0.03-1.660.660.08
United States0.931.541.190.330.980.60
Inflation
Canada0.511.360.220.501.591.84
France0.950.180.811.192.661.27
Germany-0.59-0.360.48-0.841.290.53
Italy2.150.38-1.261.852.15-0.01
Japan-1.521.38-0.86-1.01-0.11-0.32
United Kingdom1.04-2.261.051.821.400.96
United States-0.500.38-1.25-0.521.151.33
Post-1990 Sub Sample
Canada-1.01-1.170.83-2.300.681.23
France-1.020.762.10-2.300.721.88
Germany-0.490.660.61-0.550.190.03
Italy2.502.212.482.362.783.11
Japan-3.52-0.890.17-3.20-0.56-0.03
United Kingdom-0.02-2.13-1.31-0.030.611.62
United States-1.110.472.24-4.820.881.27
Current Account
Canada0.480.341.020.883.192.30
France0.112.34-0.66-0.312.94-0.56
Germany0.001.00-0.62-0.053.560.15
Italy-0.68-0.73-1.83-0.851.80-0.50
Japan0.09-0.27-2.33-0.021.48-1.46
United Kingdom0.630.20-0.541.483.41-0.17
United States-0.38-0.80-0.09-0.760.600.95
Post-1990 Sub Sample
Canada0.54-2.17-1.041.15-1.14-1.95
France0.55-0.22-1.830.091.08-0.50
Germany-0.483.871.18-0.772.571.60
Italy-1.121.63-0.56-0.650.79-1.13
Japan-0.39-1.660.32-0.32-0.33-0.49
United Kingdom1.41-2.510.023.400.151.95
United States-0.080.802.20-0.401.722.29
Import Volume
Canada1.68-0.24-0.751.480.67-0.18
France-0.35-1.06-2.05-0.20-0.42-1.50
Germany-0.93-0.54-0.57-0.280.35-0.24
Italy-0.68-0.55-1.63-1.13-0.22-1.44
Japan-0.54-1.01-1.44-0.630.39-1.21
United Kingdom0.72-0.420.680.320.570.75
United States1.63-0.43-0.161.60-0.32-0.05
Post-1990 Sub Sample
Canada1.080.170.620.98-1.10-0.47
France-1.20-0.830.18-1.05-0.86-0.64
Germany-0.83-0.23-0.26-0.090.480.81
Italy-0.181.341.21-1.46-0.88-1.18
Japan-0.162.641.82-0.190.25-1.36
United Kingdom-0.13-0.860.20-0.02-1.38-0.18
United States1.063.012.111.61-0.58-0.91
Export Volume
Canada1.290.31-0.961.691.020.51
France-0.19-3.01-2.17-0.81-1.57-1.48
Germany-0.52-1.16-1.250.06-1.11-1.30
Italy-0.89-0.65-2.23-0.960.77-0.28
Japan0.94-1.39-3.270.74-1.79-3.35
United Kingdom-0.28-2.71-0.84-0.47-0.01-1.33
United States0.39-2.97-1.420.640.03-0.20
Post-1990 Sub Sample
Canada0.781.532.240.810.670.78
France-0.36-0.622.03-0.76-0.42-1.09
Germany-1.211.072.30-0.350.90-0.23
Italy-0.881.010.58-1.30-1.40-1.38
Japan0.122.092.94-0.220.830.00
United Kingdom-0.570.041.65-1.14-3.29-2.40
United States-0.162.741.35-0.09-5.37-5.73
Sources: Author’s calculations.

t-statistics for the constant α^ in equation (1).

t-statistics for the coefficient β^ in equation (2).

t-statistics for the lagged realization in a regression of the forecast error on the lagged realization.

Sources: Author’s calculations.

t-statistics for the constant α^ in equation (1).

t-statistics for the coefficient β^ in equation (2).

t-statistics for the lagged realization in a regression of the forecast error on the lagged realization.

To see if the (relative) performance of the WEO forecasts has changed over time, we divided the sample into two equally long subintervals. The results suggest that the RMSFE-performance of the current-year and next-year WEO forecasts (relative to the naive forecasts) has worsened in most cases, sometimes by quite a significant amount. For example, in the second subsample the recursive mean now produces lower RMSFE values than the WEO next-year forecasts for France, Germany, and Italy. This could be related to the lower global volatility of GDP growth which tends to favor more smooth forecasts such as the recursive mean estimate. As is evident from the table, it has become more difficult to beat the simple benchmark forecasts during the subsequent decade.

Efficiency tests based on univariate regressions of forecasts errors on (1) a constant; (2) a constant and the lagged error; (3) a constant and the lagged realization are reported in Table 14. The first three columns present the current-year t-statistics for the constant, the lagged error and the lagged realization, while columns four through six show results for the next-year forecasts. Standard errors are heteroskedasticity and autocorrelation consistent using a Bartlett kernel with up to 2 lags, c.f. Newey and West (1987). The most notable finding is that the positive bias found in next-year forecast errors is significant for two of the countries, namely France and Germany.11

Table 14.Consensus Forecasts: Data Coverage(Number of observations)
Real GDPInflationCurrent Account
Current-

Year
Next-

Year
Current-

Year
Next-

Year
Current-

Year
Next-

Year
G-7 Countries (March/September)
Canada141314131413
France141314131413
Germany141314131413
Italy141314131413
Japan141314131413
United Kingdom141314131413
United States141314131413
Latin America (February/August)
Argentina111011101110
Brazil141314131413
Chile111011101110
Colombia111011101110
Mexico141314131413
Peru111011101110
Venezuela111011101110
Asia (March/September)
China989898
Hong Kong SAR131213121212
India989898
Indonesia131213121312
Korea141314131413
Malaysia131213121312
Singapore131213121312
Taiwan Province of China141314131413
Thailand131213121312
Source: Author’s calculations.
Source: Author’s calculations.

B. Inflation Forecasts

Inflation is generally viewed as one of the most difficult macroeconomic time series to predict, perhaps due to the evidence of instability, related to the effects of changes in monetary policy. However, for most G-7 countries the RMSFE values for inflation shown in Table 12 are quite similar to the values for GDP growth. As expected the mean absolute forecast errors are higher for the next ear than for the current-year forecasts. Furthermore, there is now evidence of positive serial correlation at the next-year horizon for six of seven economies.

Theil statistics for the ratio of RMSFE-values of the current-year WEO forecasts relative to the two naive forecasts lie between 0.36 and 0.74 (relative to the random walk forecasts) and are even lower relative to the recursive mean forecast against which they range between 0.20 and 0.37, with the lowest values observed for the United States, c.f. Table 13. Turning to the next-year forecasts, all Theil statistics remain well below unity. The relative performance of the WEO next-year forecasts is only marginally worse than the current year performance. For this variable there is systematic evidence that the WEO forecasts perform worse over time relative to the random walk forecasts, but improve relative to the recursive mean forecasts. This could well be related to structural breaks in the underlying inflation series.

Efficiency and Mincer-Zarnowitz regressions (the former shown in Table 14) only indicate scant evidence of biases in the current-year inflation forecasts, an exception being Italy.

However there is some evidence of serial correlation in the forecast errors for France, Italy, and the United Kingdom.

C. Import and Export Volume, Current Account

For the current account balance forecasts, the best naive forecast clearly comes from the random walk model and the performance against this benchmark deteriorates somewhat in the second half of our sample (and exceeds unity in one case).

Theil statistics for the WEO current- and next-year forecasts of import and export volume compared to random walk and recursive mean forecasts were all less than unity c.f. Table 13. Furthermore, the subsample analysis again shows that it has become increasingly difficult over time to beat the naive benchmarks. For both trade volume variables there are three or four instances where the RMSFE ratio exceeds unity when measured against the recursive mean forecasts. This suggests that a simple smooth forecast would perform well for around half of the countries. When measured against the random walk forecast, the RMSFE ratio is always below unity and the WEO forecasts dominate.

D. Performance After 1990

A separate analysis of the WEO forecasting performance after 1990 is warranted due to the ongoing debate on whether the productivity of the US economy has undergone a structural break and whether similar permanent or highly persistent structural changes could have affected economies such as Germany (due to the reunification in 1989) and Japan. To explore this issue we computed separate results for the post-1990 period. The outcome is reported in Table 14. There is evidence from the next-year forecasts that the WEO systematically overpredicted economic growth after 1990 for all the European G-7 economies, i.e. France, Germany, Italy, and (to a smaller degree) the United Kingdom. Japanese growth was also overpredicted after 1990 and the associated t-values are very high for most economies despite the short sample.

In contrast, US growth was underpredicted after 1990 although there is no corroborating statistically significant evidence. The same is true for the tests for serial correlation which generally fail to have sufficient power—this is unsurprising in light of the short sample used in this subsample analysis.

A similar analysis of inflation rates after 1990 shows some very interesting patterns in the next-year forecast errors. Although next-year inflation forecasts do not appear to be biased over the full sample from 1973–2003, the story is very different for the post-1990 sample. Inflation was strongly and significantly overpredicted for Canada, France, Japan, and the United States and was only underpredicted (again at a significant margin) for Italy. Furthermore, there is evidence of serial correlation in the forecast errors for Italy.

These findings have at least two possible explanations. One possibility is that the output growth and inflation series have been subject to structural breaks. This is a real possibility given the systematic evidence of structural instability underlying many reduced form models used for forecasting, c.f. Stock and Watson (1996). Another possibility is that the conditioning information underlying some of these models—such as an assumption that the output gap will be eliminated over a reasonably short span of time—gives rise to biases.

XI. Comparison of WEO and Consensus Forecasts

A comparison of forecasts to subsequent outcomes—as we have proceeded with so far—is an important exercise or “reality check” that allows us to test if basic efficiency properties are satisfied by the forecasts. This exercise clearly has its own limitations, however. For example, it is not evident what constitutes a good forecast in absolute terms. Some series may be intrinsically very difficult to predict (inflation comes to mind) because they are affected by large exogenous shocks and/or shifts in economic policy whose effect are difficult to predict in advance. Conversely, a forecast can be very uninformative but lead to errors that do not appear to violate efficiency properties such as unbiasedness and absence of serial correlation.

To address this issue, it is therefore highly informative to compare the WEO forecasts to alternative forecasts such as those produced by a highly reputed source such as the Consensus forecasts. Forecasters included in the Consensus survey faced similar difficulties as the WEO forecasters—e.g., the higher than expected productivity growth for the United States economy or the absence of large, global inflationary shocks during most of the 1990s—and therefore serve as a yardstick against which the WEO forecasts can be measured.

A second advantage of the Consensus data is that it allows us to address whether a forecaster could have done better by using both the WEO and Consensus forecasts. This is highly relevant for followers of both sets of forecasts and is ultimately also relevant for the WEO exercise—if there exists a simple combination of the WEO and Consensus forecasts that improves upon the WEO forecasts, the WEO forecasts should either be based on this combination or the potential source of inefficiency in the WEO forecasts should be identified with a view toward improving them.

This discussion motivates a comparison of the WEO forecasts with the Consensus forecasts. This is clearly a useful exercise: The Consensus forecasts constitute readily available public information so the WEO forecasts must be justified against this alternative source of forecasting information.

A. Consensus Data

To investigate the relative performance of the WEO and Consensus forecasts, we obtained Consensus forecast data on GDP growth, inflation and the current account balance over the period 1990–2003. The data covers all the G-7 economies, seven Latin American economies (Argentina, Brazil, Chile, Colombia, Mexico, Peru, and Venezuela) and nine Asian economies (China, Hong Kong, India, Indonesia, Malaysia, Singapore, South Korea, Taiwan Province of China, and Thailand).

In the baseline scenario, Consensus forecasts are measured in March (in the case of the current-year forecasts) and September (in the case of next year forecasts) except for the Latin American economies where data coverage is limited for these months. For this reason the February and August Consensus forecasts were used for these economies. The Consensus forecast is computed as the mean forecast across participants in a given monthly survey.

We shall refer to the March current-year Consensus forecast as y^t,tcons while the September next-year Consensus forecast is denoted y^t+1,tcons. Although Consensus forecasts are now available on a monthly basis, the March and September Consensus forecasts are the forecasts that are based on information whose timing is most similar to the WEO April current-year (denoted y^t,tWEO) and September next-year (y^t+1,tWEO) forecasts so this comparison was deemed most appropriate in relation to measuring the information content of the two sets of forecasts. For completeness we shall later report the outcome of a sensitivity analysis that changes the timing of the Consensus forecasts.

Table 15 shows the data coverage. The data is complete for the G-7 economies where 14 current-year forecasts and 13 next-year forecasts (1991–2003) are available. The data is somewhat less complete for Latin America where only Brazil and Mexico have full coverage while countries such as Argentina, Chile, Colombia, Peru, and Venezuela only have 10 or 11 forecasts. In the Asia region the data coverage is excellent for most of the countries except for China and India where only 8 or 9 forecasts are available.

Table 15.Correlations Between WEO and Consensus Forecasts
Forecast ErrorsForecast Levels
Current-YearNext-YearCurrent-YearNext-Year
GDP Growth
G-7 countries
Canada0.9760.9820.9850.931
France0.9500.9880.9760.939
Germany0.9460.9700.9670.838
Italy0.9670.9850.9640.893
Japan0.9650.9680.9710.908
United Kingdom0.9050.9670.9770.833
United States0.9500.9670.9780.745
Latin America
Argentina0.9120.9950.9500.901
Brazil0.9560.9160.9270.662
Chile0.6290.9710.6620.842
Colombia0.9790.9710.9390.831
Mexico0.9850.9820.9620.465
Peru0.8450.9850.4640.719
Venezuela0.9680.9770.9830.850
Asia
China0.8330.8670.9260.920
Hong Kong SAR0.9130.9810.8250.954
India0.9840.9590.9030.694
Indonesia0.9860.9850.9920.952
Korea0.9930.9960.9900.989
Malaysia0.9860.9920.9750.975
Singapore0.9760.9960.9500.976
Taiwan Province of China0.0970.9800.9550.919
Thailand0.9920.9940.9940.971
Inflation
G-7 countries
Canada0.8280.9260.9810.961
France0.8230.9600.9660.980
Germany0.8870.9040.9870.946
Italy0.9090.9400.9900.976
Japan0.7900.9240.9800.970
United Kingdom0.8920.8810.9780.976
United States0.9470.9320.9920.960
Latin America
Argentina0.3150.9910.9620.995
Brazil0.4590.9570.9080.992
Chile-0.0440.7300.9670.976
Colombia0.8590.7400.9860.976
Mexico0.6740.9660.9370.952
Peru-0.927-0.1620.8860.980
Venezuela0.8990.9350.9420.939
Asia
China0.8270.9820.9980.985
Hong Kong SAR0.8870.4960.9860.829
India0.9210.9460.8740.913
Indonesia0.9390.9960.9870.984
Korea0.9090.9210.9680.892
Malaysia0.4680.7330.7690.723
Singapore0.7560.7240.8780.406
Taiwan Province of China0.9220.6550.9590.819
Thailand0.9100.8120.9820.819
Current Account Balance
G-7 countries
Canada0.9400.9470.9710.968
France0.9570.8730.9900.935
Germany0.9590.9440.9740.892
Italy0.8840.9490.9560.954
Japan0.9280.9680.9640.971
United Kingdom0.9240.9330.9380.928
United States0.7970.8830.9920.991
Latin America
Argentina0.8790.9220.9750.957
Brazil0.9280.9840.9750.988
Chile0.9740.9840.9700.944
Colombia0.9000.9680.8660.901
Mexico0.8530.9650.9250.941
Peru0.9270.9500.9510.924
Venezuela0.9220.9420.9090.890
Asia
China0.9520.9430.9400.927
Hong Kong SAR0.8860.6870.9000.613
India0.9240.8920.9100.837
Indonesia0.9390.9370.9900.980
Korea0.9880.9900.9910.985
Malaysia0.9380.9640.9680.965
Singapore0.6690.8590.9410.952
Taiwan Province of China0.7550.9020.8600.633
Thailand0.9480.9690.9900.985
Source: Author’s calculations.
Source: Author’s calculations.

As a first check of the quality of the data, we report in Table 16 pairwise correlations between current-year (etWEO=yty^t,tWEO and etcons=yty^t,tcons) and next-year (et+1,tWEO=yt+1y^t+1,tWEO and et+1,tcons=yt+1y^t+1,tcons) forecast errors as well as the correlation between the time-series of the forecasts levels ((y^t,tWEO,y^t,tcons) or (y^t+1,tWEO,y^t+1,tcons)). In all countries and for both current year and next year forecasts and forecast errors the WEO and Consensus values are strongly positively correlated. Interestingly, the weakest correlation (0.75) is found for United States next year forecast levels. Estimated correlations are generally highest for current-year forecasts (where they all exceed 0.96) compared to next year forecasts.

Table 16.Biases in WEO and Consensus Forecasts(Mean forecast errors)
Current-yearNext-year
WEOConsensusWEOConsensus
Real GDP
G-7 countries
Canada-0.214-0.173-0.580-0.520
France-0.202-0.271-0.800-0.685
Germany-0.127-0.045-1.117-0.856
Italy-0.429-0.469-1.020-0.956
Japan0.0120.060-1.1270.834
United Kingdom-0.023-0.132-0.476-0.448
United States0.1750.2160.2310.132
Latin America
Argentina-0.165-0.956-2.461-2.406
Brazil0.017-0.049-0.598-0.983
Chile-0.332-0.719-0.949-0.963
Colombia-0.996-0.909-1.354-1.685
Mexico-0.340-0.504-1.880-1.480
Peru0.9501.601-0.535-0.099
Venezuela-0.827-1.932-3.783-3.514
Asia
China0.6090.2110.680-0.228
Hong Kong SAR-0.503-0.056-0.548-0.185
India0.3030.4790.011-0.551
Indonesia-0.161-0.342-1.546-1.544
Korea0.6380.469-0.264-0.308
Malaysia-0.109-0.202-0.542-0.658
Singapore0.337-0.0240.336-0.305
Taiwan Province of China-0.353-0.526-0.611-0.702
Thailand-0.492-0.441-1.312-1.236
Inflation
G-7 countries
Canada0.002-0.088-0.202-0.340
France0.0430.013-0.051-0.284
Germany0.0910.0580.116-0.030
Italy0.0820.1050.2520.142
Japan-0.0700.015-0.242-0.256
United Kingdom-0.0730.075-0.256-0.365
United States0.0050.025-0.296-0.370
Latin America
Argentina-1.456-5.668-2.018-1.838
Brazil288.62172.307326.132216.726
Chile0.150-0.242-0.163-0.321
Colombia0.302-0.547-0.796-0.249
Mexico2.3341.6344.0683.680
Peru1.415-3.8170.094-3.093
Venezuela2.6632.26010.5956.671
Asia
China-1.378-2.605-3.255-3.960
Hong Kong SAR-1.051-1.174-1.610-2.356
India0.188-0.887-0.982-1.441
Indonesia1.1012.1834.5483.796
Korea-0.239-0.5010.062-0.243
Malaysia-0.491-0.679-1.144-0.916
Singapore-0.415-0.560-0.551-0.989
Taiwan Province of China-0.630-0.743-0.814-0.983
Thailand-0.972-0.772-0.688-1.063
Current Account Balance
G-7 countries
Canada1.1191.8743.2951.854
France1.2854.1250.5825.389
Germany-2.463-0.770-5.078-4.075
Italy-3.609-2.151-3.430-2.556
Japan-2.1718.6521.28711.024
United Kingdom3.0841.7557.0357.707
United States-0.461-8.706-9.695-20.276
Latin America
Argentina0.212-0.1101.5911.791
Brazil-1.346-1.351-0.1820.532
Chile0.4600.3990.8610.722
Colombia0.061-0.0610.0300.296
Mexico-1.5920.502-0.3221.225
Peru-0.0090.0890.2460.130
Venezuela3.0653.4773.7363.552
Asia
China7.2117.21617.69616.389
Hong Kong SAR1.2382.8050.1142.854
India1.8530.9744.4913.725
Indonesia1.2020.8022.9301.588
Korea1.0360.8103.0342.164
Malaysia1.4222.0582.8902.877
Singapore2.4132.5482.8994.201
Taiwan Province of China1.8621.2232.9524.075
Thailand2.2471.5383.6212.384
Source: Author’s calculations
Source: Author’s calculations

For the inflation variable, forecast errors and the forecasts themselves are again strongly positively correlated among the G-7 and Asian countries. However, in some Latin American countries that were affected by very high levels of inflation during the sample, the forecast errors are only weakly or even negatively correlated as in the case of Chile and Peru. Forecast levels continue to be strongly positively correlated for all Latin American countries. Current account forecast errors and forecast levels again see strong positive pair-wise correlations between Consensus and WEO values.

B. Biases in Forecasts

Since the mean Consensus forecast is an example of a combined forecast and combined forecasts generally perform quite well, c.f. Clemen (1989), it should be noted that a comparison against the Consensus value sets the bar high—evidence that the WEO forecasts outperform the Consensus forecasts will essentially have to overcome the expected gain from combining.

Table 17 reports the estimated bias in the WEO and Consensus forecasts. Whereas the current-year WEO forecasts are generally less biased than the current-year Consensus forecasts, for the next-year values the bias is stronger in the WEO forecasts than in the Consensus forecasts for every single country.

Table 17.Comparison of WEO and Consensus Forecasts: Sign of Forecasts and Forecast Differentials(Fraction of positive forecast errors and differences)
Current-yearNext-yearDifferences between

WEO and Consensus
WEOConsensusWEOConsensusCurrent-

Year
Next-year
Real GDP
G-7 Countries
Canada0.3570.3570.3850.3850.5000.692
France0.3570.2860.3080.3850.5000.615
Germany0.5000.5000.1540.2310.6430.846
Italy0.2860.2860.2310.2310.4290.769
Japan0.5000.5710.3080.3080.4290.692
United Kingdom0.3570.2860.3080.3850.4290.615
United States0.6430.6430.6150.5380.5710.308
Latin America
Argentina0.5450.3640.4000.4000.3640.600
Brazil0.3570.3570.3850.3080.5000.462
Chile0.4550.3640.3000.4000.4550.600
Colombia0.2730.4550.3000.4000.5450.400
Mexico0.5000.4290.3080.3080.3570.769
Peru0.4550.6360.5000.5000.6360.700
Venezuela0.2730.3640.3000.3000.1820.500
Asia
China0.7780.5560.6250.5000.2220.000
Hong Kong0.4620.6150.5000.5000.6920.583
India0.6670.6670.5000.3750.8890.000
Indonesia0.7690.6920.5000.4170.3850.417
Korea0.5000.5000.4620.4620.3570.538
Malaysia0.6150.6920.6670.4170.5380.250
Singapore0.4620.4620.5000.5000.2310.083
Taiwan Province of China0.4290.3570.5380.3850.2860.538
Thailand0.5380.4620.5000.5000.5380.500
Inflation
G-7 Countries
Canada0.2860.2860.3080.4620.3570.308
France0.6430.5000.4620.3080.4290.077
Germany0.6430.4290.5380.3850.4290.308
Italy0.7140.6430.4620.5380.5710.385
Japan0.3570.5710.3080.2310.5710.462
United Kingdom0.4290.4290.1540.0770.6430.385
United States0.4290.4290.1540.1540.3570.385
Latin America
Argentina0.2730.0000.1000.1000.3640.600
Brazil0.7140.7140.6920.6920.7140.308
Chile0.5450.5450.5000.5000.4550.400
Colombia0.5450.4550.3000.5000.2730.400
Mexico0.7860.7860.7690.5380.3570.308
Peru0.3640.1820.4000.2000.1820.000
Venezuela0.6360.4550.7000.7000.4550.200
Asia
China0.2220.1110.0000.1250.2220.250
Hong Kong SAR0.1540.1540.1670.0830.3850.417
India0.3330.3330.1250.1250.1110.375
Indonesia0.4620.5380.5830.5830.5380.333
Korea0.5000.3570.4620.3850.2140.308
Malaysia0.3850.1540.2500.1670.4620.583
Singapore0.3850.2310.3330.0000.3850.333
Taiwan Province of China0.2140.2140.3080.2310.5000.462
Thailand0.3080.3080.4170.3330.5380.333
Current Account Balance
G-7 Countries
Canada0.5710.7140.6150.4620.6430.308
France0.5710.7860.6150.7690.9290.846
Germany0.3570.4290.4620.3850.6430.462
Italy0.2860.3570.3080.3080.5000.538
Japan0.5000.7140.5380.6150.9290.923
United Kingdom0.6430.5710.7690.7690.2140.538
United States0.4290.3570.3850.2310.2860.308
Latin America
Argentina0.5450.4550.5000.5000.5450.300
Brazil0.2860.3570.5380.5380.5000.538
Chile0.6360.6360.7000.7000.4550.300
Colombia0.3640.3640.5000.5000.4550.600
Mexico0.4290.3570.3850.4620.7860.692
Peru0.6360.6360.6000.6000.7270.400
Venezuela0.7270.7270.8000.8000.6360.500
Asia
China0.6670.7780.8750.7500.5560.500
Hong Kong SAR0.5830.6670.5830.7500.6670.750
India0.7780.5560.8750.8750.3330.250
Indonesia0.7690.7690.7500.6670.2310.083
Korea0.3570.5000.4620.3850.3570.308
Malaysia0.6150.6920.7500.8330.6150.417
Singapore0.6920.9230.8330.7500.6150.583
Taiwan Province of China0.6430.5710.6920.6920.5000.615
Thailand0.6150.6920.6670.5830.3080.167
Source: Author’s calculations.
Source: Author’s calculations.

Similar findings hold for Latin America—the WEO forecasts generate smaller biases than the Consensus forecasts of the current-year values, whereas the bias is very similar for the two sets of next-year forecasts. In many cases the biases in both sets of forecasts are large in economic terms—the most extreme case being Venezuela which has a bias of -3.8 percent in the next-year forecast. Even for the G-7 economies biases of over 1 percent per year are observed for three economies—in all three cases (Germany, Italy, and Japan) the WEO overpredicted growth.

In Asia the overall WEO versus Consensus forecasting performance is quite similar for the current-year and next-year forecasts. There are some interesting differences among the individual countries, however. The WEO forecasts of growth in China are generally downward biased compared to the Consensus forecasts while conversely the forecasts for Hong Kong were upward biased giving rise to a large negative mean in the forecast error.

The (absolute) value of the bias in both WEO and Consensus forecasts is higher in the next-year than in the current-year forecasts. This holds for most countries and variables and is what one would expect to find by random chance given that the next-year forecast errors have higher variance than current-year forecast errors so the next-year sample mean is more dispersed than the current year sample mean.

The bias in next-year inflation forecasts is also higher than in the current year forecasts in the majority of the G-7 and Asian economies. In economic terms, however, this bias is not particularly large for the G-7 economies where all the WEO values fall below 0.3 percent, while the Consensus values fall below 0.4 percent. Biases are greater in Asia and Latin America in particular, reflecting the much higher levels of inflation experienced in these regions during the nineties. WEO biases for Latin America are predominantly positive suggesting a tendency to underpredict inflation for these economies. Conversely, both the WEO and Consensus forecast errors have negative means for the Asian economies, suggesting overprediction of inflation in this region. Notice that the WEO forecasts perform better than the Consensus forecasts in terms of the absolute value of the bias for eight out of nine current-year forecasts and for seven out of nine next year forecasts of inflation in Asia.

Both the Consensus and WEO forecasts systematically underpredict the current account in Asia. Furthermore, the Consensus forecast underpredicted the next year current account for every single country in Latin America.

To see whether the WEO and Consensus forecasts differ systematically in the sense that one forecast is generally higher or lower than the other, Table 18 shows the percentage of positive current and next year forecast errors for the WEO and Consensus (first four columns) in addition to the proportion of positive forecast differentials (y^t,tWEOy^t,tcons and y^t+1,tWEOy^t+1,tcons) in columns five and six. If differences between the two sets of forecasts were purely random and symmetric, one would expect this percentage to be close to 0.50, where closeness is of course related to the sampling distribution of this statistic.12 In most cases differences between the two sets of forecasts are not systematic with a sign proportion close to one-half, although there are some exceptions, e.g., next year GDP forecasts for Germany, China, India, and Singapore, next year inflation forecast for France and Peru and current account balance forecasts for France, Japan, and Indonesia (where the WEO forecasts are systematically higher than the Consensus values).13

Table 18.Comparison of WEO and Consensus Forecasts: Significance of Biases and Serial Correlations(Bootstrapped p-values)
Current-yearNext-year
WEOConsensusWEOConsensus
Bias 1/Joint 2/Bias 1/Joint 2/Bias 1/Joint 2/Bias 1/Joint 2/
Real GDP
G-7 countries
Canada0.4600.7720.5330.6270.2320.4800.2760.637
France0.2570.2700.1930.2020.0530.1750.0840.299
Germany0.6060.3970.8710.5610.0200.0230.0530.163
Italy0.0740.2460.0450.1410.0060.0250.0100.044
Japan0.9760.9780.8770.9240.0530.1140.1380.289
United Kingdom0.9060.8430.5810.835-0.2340.1680.2960.192
United States0.3730.3720..2830.3280.6150.4370.7670.691
Latin America
Argentina0.8940.9430.4600.7710.2660.3020.2810.338
Brazil0.9770.8310.9530.8910.3400.3160.1440.300
Chile0.5540.4030.2260.5560.2570.4110.2550.455
Colombia0.1480.1930.2240.2360.1180.1880.1040.154
Mexico0.5810.7990.4680.6540.1040.2070.1870.310
Peru0.3790.5280.2210.3830.6860.3660.9430.573
Venezuela0.5390.3720.1670.4440.0390.1790.0500.216
Asia
China0.0530.2220.4990.8370.1300.3770.5800.684
Hong Kong SAR0.5930.5550.9540.6700.6490.6400.8970.782
India0.4460.7800.2960.7030.9770.9610.2730.409
Indonesia0.8550.9490.6830.7960.4540.6200.4500.604
Korea0.4720.5480.5790.5510.8660.5640.8450.510
Malaysia0.9140.5250.8350.4820.7090.5300.6590.529
Singapore0.7100.8130.9760.9230.7850.9630.8070.986
Taiwan Province of China0.5190.6020.4050.5010.4520.5220.4280.449
Thailand0.5550.4740.5750.6150.3850.6630.3600.584
Inflation
G-7 countries
Canada0.9870.6700.4800.2910.3140.5880.1840.381
France0.6390.5090.8800.0450.7750.0720.1130.017
Germany0.4050.4640.6530.3860.6170.4420.8920.407
Italy0.6060.9410.5060.8670.3480.6400.5030.789
Japan0.5670.3690.9030.7760.2130.1840.0840.106
United Kingdom0.6890.4890.7240.5800.1200.3790.0150.041
United States0.9620.8860.8260.8910.0710.1900.0390.097
Latin America
Argentina0.2650.1920.1200.1980.7990.0240.7560.024
Brazil0.1280.1400.8160.7000.1160.0450.1940.126
Chile0.5070.9040.4440.6930.6370.0090.2450.300
Colombia0.4930.4710.3920.7530.3280.1140.7510.513
Mexico0.0160.0060.1590.2330.3370.4260.2540.320
Peru0.5130.3970.3620.2440.9080.8130.0980.104
Venezuela0.2820.5630.5670.7010.0680.2700.2420.537
Asia
China0.0900.0940.0500.0440.0210.0120.0230.018
Hong Kong SAR0.0970.0060.0280.0020.1770.0120.0010.001
India0.8240.4720.5160.1630.4500.6690.2040.387
Indonesia0.5170.2950.4510.5110.4360.4580.4710.469
Korea0.5720.4280.2860.2010.9150.2300.6070.309
Malaysia0.1330.0520.0090.0100.0470.0540.0090.027
Singapore0.1580.3410.0590.0360.1210.1970.0030.013
Taiwan Province of China0.0540.1410.0160.0210.0180.0220.0050.020
Thailand0.0410.1100.0730.1700.3210.3890.1250.315
Current Account
G-7 countries
Canada0.6070.7200.3510.4860.2550.0970.5220.290
France0.6110.3470.0980.1890.8710.0640.1640.096
Germany0.6530.5590.9030.1940.5300.2770.6370.071
Italy0.3070.5290.5380.5210.5680.4110.6490.272
Japan0.7150.8160.1460.1920.8790.4850.1720.396
United Kingdom0.1950.4460.4430.5960.0110.0450.0190.091
United States0.9340.2970.3750.5680.5350.4070.2160.161
Latin America
Argentina0.7960.3940.9240.6760.5210.6510.4650.567
Brazil0.4890.0190.4930.0280.9570.0830.8650.094
Chile0.3170.3690.4260.4680.1640.4460.2740.570
Colombia0.8740.8740.9210.7240.9630.3380.6990.185
Mexico0.2620.1690.7600.8380.9100.6100.6900.560
Peru0.9670.9170.7280.2580.4770.1290.7300.302
Venezuela0.0300.0480.0240.0590.0190.0900.0250.090
Asia
China0.0980.1780.0520.1390.0720.1930.0310.054
Hong Kong SAR0.4470.5470.0700.1390.9450.6790.1150.238
India0.1170.2310.4280.2480.0080.0120.0230.004
Indonesia0.0560.0580.2160.0750.0430.0510.2730.467
Korea0.7060.8300.7840.7680.5500.6810.6700.777
Malaysia0.2710.3470.1510.1860.1080.0710.0940.120
Singapore0.0710.1490.0170.0370.0660.1670.0210.072
Taiwan Province of China0.1920.3490.4640.0300.1950.2440.0650.030
Thailand0.1200.1660.2030.2080.1250.2160.2990.320
Source: Authors’ calculations.

Bootstrapped p-values for the student t-statistics of the bias coefficient α^ in equation (1).

Bootstrapped p-values for the F-test statistics of the joint test α^=0, β^=0 in equation (2).

Source: Authors’ calculations.

Bootstrapped p-values for the student t-statistics of the bias coefficient α^ in equation (1).

Bootstrapped p-values for the F-test statistics of the joint test α^=0, β^=0 in equation (2).

C. Uncovering Sources of Relative Performance

As a further step towards understanding the source of the relative performance of the WEO visà-vis the Consensus forecasts, we bootstrapped p-values for the student-t statistic associated with tests for bias in the forecasts and an F-test for a non-zero bias and first-order serial correlation based on equation (2). Results are shown in Table 19. This table contains eight columns reporting the p-values for four pairs of tests for the current year WEO and Consensus forecasts and the next year WEO and Consensus forecasts. Values below 0.05 indicate statistical significance.

Table 19.Comparison of WEO and Consensus Forecasts: Ratios of Root-Mean-Squared Forecast Errors(Consensus over WE0; Consensus measured in March/September)
GDPInflationCurrent Account
Current-

Year
Next-

Year
Current-

Year
Next-

Year
Current-

Year
Next-

Year
G-7 Countries
Canada0.9920.9770.9451.2670.9071.000
France1.1500.9481.0881.0741.0521.070
Germany1.0920.9061.1550.9861.1021.093
Italy0.9441.0140.9950.7831.0000.909
Japan1.0530.9261.0810.7851.0341.034
United Kingdom1.1841.0241.1940.9240.9731.177
United States1.0260.9650.9621.0171.5461.065
Latin America
Argentina1.0360.9983.3330.9091.2990.985
Brazil1.1211.0851.5990.7540.9761.035
Chile1.0981.0141.2890.7871.0961.069
Colombia1.1271.1861.4820.9121.4931.234
Mexico1.1070.9881.2070.9951.1161.052
Peru1.2720.9901.3661.9221.2041.012
Venezuela1.1000.9631.6170.9281.0980.962
Asia
China0.9150.8531.5221.2460.8810.798
Hong Kong SAR0.9881.0960.9100.8201.0241.086
India1.1531.0661.3780.9351.0070.921
Indonesia0.9511.0111.6101.0010.9890.884
Korea0.9580.9631.1060.7861.0851.010
Malaysia0.9411.0050.7670.7161.0810.986
Singapore0.9821.0421.0711.0420.8741.164
Taiwan Province of China1.1050.9790.9431.0051.1540.992
Thailand0.9540.9020.9321.1100.8840.985
Source: Author’s calculations.
Source: Author’s calculations.

In the majority of cases there is no significant evidence of bias and/or serial correlation in the forecast errors. However, there are some interesting exceptions. For example, the next year WEO forecasts of GDP growth in Germany and Italy appear to be strongly biased and serially correlated and there is further evidence of a bias for France and Japan. The Consensus forecasts seem to have had similar problems producing unbiased forecasts for Germany and Italy although serial correlation only is a problem for the latter economy.

Continuing to the Latin American and Asian GDP forecasts, there is essentially no evidence of bias or serial correlation—with the possible exception of a bias in the next-year WEO and Consensus forecasts for Venezuela.

The inflation forecasts tell a very different story. There are now few significant cases of bias or serial correlation for the G-7 economies—the main difference being some serial correlation and/or bias in the next-year Consensus forecasts for France, the United Kingdom and the United States. There are also few significant p-values for Latin America—current-year WEO forecasts for Mexico being one notable exception. The most notable difference is that there is a relatively large number of rejections of efficiency for the inflation forecasts in Asia. Both the WEO and Consensus forecasts appear to have been biased and/or serially correlated at the current year and next-year horizons for China, Hong Kong, Malaysia, and Taiwan. In addition the Consensus inflation forecasts are also biased and serially correlated for Singapore. Current account forecasts were mostly unbiased and the forecast errors serially uncorrelated with a few exceptions such as the WEO current-year forecast for Brazil (which appears to be serially correlated) and next year forecast for the United Kingdom (biased and serially correlated). Both the WEO and Consensus next-year forecasts for India were biased and serially correlated as were the current and next year forecasts of Venezuela’s current account balance.

D. Statistical Tests of Forecasting Performance

To gain a more precise evaluation of the relative performance of the two sets of forecasts, Table 19 shows the ratio of Consensus over WEO root mean squared forecast errors (RMSFE). Values below one suggest that the Consensus forecast performed best over the sample, while values above one suggest that the WEO forecasts were better.

Current-year GDP forecasts produced by the WEO are on average better than the Consensus forecasts for the G-7 and Latin American economies as the RMSFE ratio exceeds unity for five of seven G-7 countries and for all Latin American countries—although it should be borne in mind that the current and next-year forecasts for the latter countries are measured in February and August, respectively. In contrast, the Consensus forecasts are better for the Asian economies as only two of nine current year RMSFE ratios exceed unity. Turning to the next year GDP values, the performance of the two sets of forecasts is very similar with RMSFE ratios between 0.90 and 1.10 in all but two cases. One notable exception is for China where the WEO forecast is notably worse than the Consensus forecast as witnessed by the RMSFE ratio of 0.85.

The WEO current-year inflation forecasts perform quite well relative to the Consensus values in all three regions, particularly in Latin America. Conversely, next-year inflation forecasts produced by the Consensus survey are generally better than the WEO next-year forecasts for Latin America (with the exception of Peru) while the two sets of forecasts are of similar quality for the G-7 and Asian economies.

The WEO current year and next year forecasts of the current account balance are generally better than the Consensus values in the G-7 and Latin American regions while the two forecasts are of similar quality in Asia.

To assess the statistical significance of differences in the performance of the Consensus and WEO forecasts, we apply the framework proposed by Diebold and Mariano (1995). Let L(.) be a loss function representing the cost of making 30 an error in forecasting a variable of interest. A common assumption is that loss only depends on the squared value of the forecast error, e, and thus takes the form L(e) = e2. To evaluate whether the average MSFE-value generated by the WEO and Consensus forecasts are genuinely different, we define the following loss differential dift+1=(et+1,tWEO)2(et+1,tcons)2. A test can now be based on the t-statistic associated with the mean of this differential, dif¯=T1r=1Tdifτ, such as dif¯/σ^(dif¯), where σ^(dif¯) is an estimate of the standard error of dif¯. This statistic can be obtained from a least squares regression of the time-series of loss differentials on an intercept. To evaluate the statistical significance of this statistic we bootstrap the sampling distribution of the mean loss differential, imposing the null of no difference by re-centering the squared loss differentials so they have a mean of zero (in accordance with the null hypothesis).

Results from this analysis are shown in Table 21. Negative numbers represent cases where the WEO forecasts generate lower MSFE values than the Consensus forecasts while positive values indicate that the Consensus forecasts were best. P-values show the percentage of bootstraps where the loss differential was greater than that observed in the actual data. Values below 0.05 therefore indicate that the Consensus forecasts are significantly better than the WEO forecasts, while p−values above 0.95 suggest that the WEO forecasts are significantly better.

Unsurprisingly in view of the small samples, not that many cases produce significant test statistics. It is interesting to note, however, that there are more significant cases where the WEO dominates the Consensus than the reverse, particularly in the case of the current-year forecasts. The WEO current-year GDP forecast is best for the United Kingdom, Colombia, Mexico, and Indonesia, while the WEO forecast dominates for Thailand. Current year WEO inflation forecasts for Argentina, Venezuela, and China dominate their Consensus counterparts, while the reverse holds for next-year inflation forecasts for Italy, Japan, Brazil, Hong Kong, Singapore, and South Korea.

The two sets of current account balance forecasts are quite similar although there are more cases where the WEO forecasts are better than the Consensus forecasts than the reverse.

E. Tests of Informational Efficiency

A question of intrinsic interest is whether the Consensus forecasts can help in predicting the WEO forecast errors. Table 21 reports regression results that set zt=y^t,tcons in the case of the current-year forecasts or zt=y^t+1,tcons (for next-year forecasts) in the efficiency regression (4) based on the current year and next year WEO forecast errors

If β = 0, then the WEO forecasts effectively encompass the information embodied in the Consensus forecasts. The estimated coefficient on the Consensus forecast exceeds two in absolute value for six of 46 of the current-year and next year WEO GDP forecast errors (Brazil, Mexico, Venezuela, China, Indonesia, and South Korea), in eight cases for the inflation forecasts and in five cases for the current account balance forecasts.

In the reverse regression of Consensus forecast errors on WEO forecasts,

the estimated coefficient on the WEO forecast was significant in explaining GDP, inflation and current account Consensus forecast errors in three, twelve and three cases, respectively. In these cases the Consensus forecasts fail to encompass the WEO forecasts.14

F. Timing of Consensus Forecasts

The information sets underlying the Consensus and WEO forecasts are not perfectly aligned, so it is of interest to investigate the sensitivity of the (relative) performance of the two sets of forecasts to changes in the dating. We do so in two ways. First, we compare the published (April/September) WEO forecasts to the Consensus forecasts reported in February and August, respectively. This timing clearly benefits the WEO forecasts which can embody more up-to-date information than is available in February or August. We also reverse the informational advantage by comparing the WEO forecasts to the April/October Consensus forecasts that embody more recent information than the WEO forecasts.

If the Consensus and WEO forecasters update their predictions reasonably efficiently, we would expect that the Consensus/WEO RMSFE ratios should be higher than the benchmark results reported in Table 20 when the Consensus forecasts are based on the February/August information, while conversely we would expect to see lower values when using the April/October Consensus forecasts.

Table 20.Comparison of WEO and Consensus Forecasts: Squared Loss Differentials
Current-Year ForecastsNext-Year Forecasts
Mean

Differential
T-StatisticsP-Value 1/Mean

Differential
T-StatisticsP-Value 1/
GDP Growth
G-7 countries
Canada0.0180.1200.4360.1350.4940.305
France-0.133-1.0710.9300.2351.2910.075
Germany-0.156-0.8100.8090.5641.1910.083
Italy0.0880.9720.150-0.053-0.3300.643
Japan-0.196-0.8760.7880.6531.1280.088
United Kingdom-0.204-1.4420.963-0.093-0.2260.622
United States-0.027-0.3240.6230.1660.4260.365
Latin America
Argentina-1.072-0.8410.8360.1800.0460.439
Brazil-1.709-1.0980.896-0.845-1.0090.864
Chile-0.634-0.5330.756-0.183-0.1230.583
Colombia-1.196-1.2820.957-2.798-1.3420.955
Mexico-1.105-1.5240.9750.3800.2690.370
Peru-6.562-0.8570.8910.2850.0890.481
Venezuela-3.754-0.7230.7692.5420.6190.238
Asia
China0.1410.5350.3480.4300.5840.285
Hong Kong SAR0.2280.1940.414-3.103-1.3570.959
India-0.438-2.2340.976-0.227-0.4820.664
Indonesia0.7931.3930.059-0.748-0.5740.747
Korea0.7491.3190.0721.8740.8600.142
Malaysia1.2060.8820.137-0.239-0.1550.539
Singapore0.3430.4330.298-1.398-0.7850.805
Taiwan Province of China-0.699-1.1820.9480.2361.2410.104
Thailand0.7220.8430.1384.5861.4550.021
Inflation
G-7 countries
Canada0.0240.3630.362-0.296-1.2590.939
France-0.019-0.6640.681-0.057-0.6590.747
Germany-0.052-0.8190.8280.0180.1030.448
Italy0.0030.0470.4940.3391.8180.008
Japan-0.032-0.4580.6780.1761.6510.034
United Kingdom-0.172-0.8370.8610.0500.6770.260
United States0.0130.3430.346-0.012-0.1830.549
Latin America
Argentina-86.000-1.5280.97632.2990.8800.072
Brazil-545,260.635-1.0710.929181,856.3191.8170.012
Chile-0.323-0.4790.7430.3820.8830.113
Colombia-2.228-1.2640.9371.0420.5440.241
Mexico-5.495-0.6700.7930.7720.0660.458
Peru-34.005-1.1300.930-17.132-1.3110.956
Venezuela-98.608-2.7580.99646.6450.6440.263
Asia
China-7.830-1.7790.975-9.205-2.1930.987
Hong Kong SAR0.7730.7580.1893.7980.8520.146
India-5.847-1.0540.9321.0770.3410.325
Indonesia-44.632-0.9230.928-0.264-0.0420.559
Korea-0.509-0.9740.8561.4831.5400.021
Malaysia0.5761.3550.0721.8901.4920.038
Singapore-0.142-0.6150.698-0.128-0.3200.618
Taiwan Province of China0.1740.5160.290-0.017-0.0440.529
Thailand0.3660.6510.198-0.898-1.8210.964
Current Account Balance
G-7 countries
Canada10.6651.1530.0940.0070.0000.477
France-8.366-0.3640.611-22.122-0.3730.617
Germany-80.653-0.8740.849-146.917-0.9980.871
Italy-0.101-0.0040.50877.1141.5370.039
Japan-31.195-0.2260.569-54.208-0.3360.617
United Kingdom3.9930.2360.399-39.351-1.8300.991
United States-697.395-1.7440.977-383.782-0.5690.718
Latin America
Argentina-4.825-1.5200.9651.6020.1510.420
Brazil2.3740.2650.384-8.346-0.8110.798
Chile-0.382-1.3560.933-0.516-0.9010.795
Colombia-1.996-1.6560.974-1.891-1.7280.958
Mexico-6.301-0.4790.724-10.490-0.7310.835
Peru-0.208-2.0700.989-0.026-0.0920.545
Venezuela-4.549-0.7260.7892.2540.4540.297
Asia
China37.5361.0790.095275.3421.9590.016
Hong Kong SAR-1.333-0.1300.528-5.969-0.3260.582
India-0.154-0.0850.5374.3090.7510.216
Indonesia0.1120.1070.4615.4541.7910.015
Korea-15.977-1.2660.956-5.390-0.5740.746
Malaysia-3.522-0.8400.8220.9230.4370.320
Singapore4.8780.4700.253-11.067-1.2070.910
Taiwan Province of China-9.080-1.0630.8680.9690.0610.480
Thailand4.6951.6280.0261.6320.3470.375
Source: Author’s calculations.

Bootstrapped p-value for the t-statistics for the mean differential.

Source: Author’s calculations.

Bootstrapped p-value for the t-statistics for the mean differential.

Table 21.Comparison of WEO and Consensus Forecast: Do Consensus Forecasts Help in Predicting WEO Forecast Errors?(WEO forecast errors projected on a constant and Consensus forecast)
Current-Year ForecastsNext-Year Forecasts
ConstantConsensus ForecastConstantConsensus Forecast
Estimatet-StatisticsEstimatet-StatisticsEstimatet-StatisticsEstimatet-Statistics
Real GDP
G-7 countries
Canada-0.359-0.5300.0570.2370.0500.025-0.210-0.321
France-0.082-0.168-0.056-0.2642.2291.175-1.212-1.625
Germany0.1130.258-0.145-0.6651.0430.727-0.981-1.560
Italy-0.435-0.7150.0030.0100.0470.030-0.469-0.690
Japan0.0100.0200.0010.006-0.192-0.182-0.513-1.023
United Kingdom0.0820.205-0.055-0.304-1.619-1.0700.4830.780
United States-0.407-0.8590.2361.338-0.208-0.1000.1700.215
Latin America
Argentina-0.310-0.2080.0600.187-1.164-0.215-0.384-0.263
Brazil1.4401.881-0.769-2.8323.1101.054-1.141-1.282
Chile-2.731-2.0910.4701.975-3.811-0.7040.5440.535
Colombia-2.363-1.3890.4100.858-0.098-0.027-0.326-0.353
Mexico-2.564-2.0690.7202.0000.7140.114-0.657-0.420
Peru0.8900.4750.0180.0389.6091.352-2.197-1.448
Venezuela-0.697-0.625-0.435-2.214-3.979-1.7010.1020.116
Asia
China3.5632.219-0.359-1.8577.6453.932-0.824-3.610
Hong Kong SAR0.2160.103-0.187-0.3802.3211.015-0.725-1.438
India7.4102.374-1.284-2.2899.5971.623-1.503-1.626
Indonesia-1.150-0.9470.2401.1070.1240.033-0.328-0.495
Korea-0.251-0.1190.1630.45710.3612.610-1.753-2.799
Malaysia-1.932-0.7940.3030.8144.0151.040-0.722-1.262
Singapore0.1110.0430.0410.0944.1541.046-0.654-1.010
Taiwan Province of China0.4780.227-0.149-0.4060.2580.073-0.149-0.253
Thailand-0.853-0.5690.0710.290-0.685-0.210-0.114-0.217
Inflation
G-7 countries
Canada0.1460.542-0.061-0.6160.2120.453-0.170-0.974
France0.0080.0320.0180.1530.6111.310-0.320-1.520
Germany-0.131-0.4760.1000.877-0.507-0.7910.2671.039
Italy0.3010.801-0.061-0.6410.7211.120-0.142-0.799
Japan-0.029-0.208-0.060-0.578-0.237-1.020-0.007-0.037
United Kingdom-0.597-1.5840.1641.549-0.110-0.217-0.045-0.303
United States0.1130.260-0.038-0.2580.2270.315-0.172-0.742
Latin America
Argentina-0.938-0.964-0.048-0.9453.7741.238-0.907-4.106
Brazil211.0261.3890.1391.302162.6771.1220.6592.684
Chile-0.468-1.2270.0971.876-0.592-0.7650.0750.618
Colombia-0.333-0.2830.0410.5800.2570.107-0.072-0.465
Mexico-1.569-1.3310.2643.6715.2400.913-0.098-0.225
Peru-2.612-2.4620.2696.450-1.511-1.4050.1532.021
Venezuela-4.081-0.7350.1741.328-2.893-0.2340.3901.191
Asia
China-1.260-1.273-0.024-0.185-0.797-1.473-0.493-6.205
Hong Kong SAR-1.714-2.2720.1381.215-3.229-2.3500.3101.508
India2.0430.805-0.256-0.784-4.468-0.9850.4760.791
Indonesia-1.905-0.7250.2681.3699.8171.267-0.540-0.836
Korea0.9110.987-0.208-1.383-0.864-0.4940.1860.562
Malaysia-0.4440.439-0.013-0.0490.3740.194-0.392-0.816
Singapore-0.365-0.714-0.024-0.116-2.049-2.3160.6671.796
Taiwan Province of China-0.845-1.2350.0740.353-2.108-3.5030.4272.340
Thailand0.4500.714-0.311-2.624-0.561-0.366-0.027-0.090
Current Account Balance
G-7 countries
Canada2.6431.0480.2091.1094.2471.3680.1740.749
France4.4291.345-0.209-1.3637.0671.635-0.459-2.163
Germany-2.586-0.494-0.259-1.253-5.989-0.985-0.995-2.946
Italy-2.487-0.684-0.142-0.8210.1190.019-0.420-1.525
Japan25.8801.295-0.298-1.46258.3642.537-0.589-2.593
United Kingdom5.5661.0820.1410.5424.4040.797-0.130-0.517
United States2.4480.2170.0130.31412.0770.4580.0971.009
Latin America
Argentina-0.492-0.454-0.117-1.018-2.418-0.786-0.558-1.804
Brazil-0.135-0.0460.0970.566-4.200-0.907-0.274-1.158
Chile-0.200-0.280-0.345-1.122-0.808-0.683-0.732-1.573
Colombia-0.638-0.914-0.255-1.212-2.031-1.570-0.639-1.775
Mexico-4.806-1.676-0.232-1.260-12.575-2.474-0.784-2.705
Peru-0.056-0.090-0.018-0.080-0.034-0.029-0.108-0.250
Venezuela3.6002.757-0.333-0.8524.8082.433-0.512-0.757
Asia
China17.4412.619-0.955-1.78246.5633.492-3.048-2.436
Hong Kong SAR1.2560.636-0.005-0.017-0.449-0.1960.1840.403
India1.1070.566-0.191-0.4556.8162.4070.4620.890
Indonesia1.2682.2210.0920.8062.9212.261-0.012-0.055
Korea0.9350.3380.0480.1853.8670.879-0.660-1.460
Malaysia1.4131.0860.0700.2872.9051.8290.0820.255
Singapore1.4910.6400.0800.4556.7402.369-0.339-1.536
Taiwan Province of China0.7460.2250.1300.371-2.179-0.3710.7010.936
Thailand2.2241.778-0.013-0.0923.1201.592-0.249-1.115
Source: Author’s calculations.
Source: Author’s calculations.

Table 22 and Table 23 present RMSFE ratios when the current- and next year Consensus forecasts are the ones published in February/August or in April/October. The February Consensus GDP current-year forecasts generate higher RMSFE values than the WEO forecasts for six, seven, and eight of the G-7, Latin American and Asian economies, respectively. On the other hand, the WEO next-year GDP forecasts only dominate the Consensus forecasts in roughly half of the cases despite the latter’s use of dated information relative to the WEO forecasts.

Table 22.Comparison of WEO and Consensus Forecasts: Ratios of Root-Mean-Squared Forecast Errors(Consensus over WEO; Consensus measured in February/August)
Real GDPInflationCurrent Account
Current-

Year
Next-

Year
Current-

Year
Next-

Year
Current-

Year
Next-

Year
G-7 countries
Canada1.1670.9651.2731.3420.9980.998
France1.3491.0241.2741.1031.0932.650
Germany1.2020.9571.3211.0131.1621.213
Italy0.9871.0661.2020.8911.0580.919
Japan1.1280.9731.0970.8581.1770.995
United Kingdom1.2911.0691.4191.0180.9771.177
United States1.1581.0311.0561.0451.5941.098
Latin America
Argentina1.0360.9983.3330.9091.2990.985
Brazil1.1211.0851.5990.7540.9761.035
Chile1.0981.0141.2890.7871.0961.069
Colombia1.1271.1861.4820.9121.4931.234
Mexico1.1070.9881.2070.9951.1161.052
Peru1.2720.9901.3661.9221.2041.012
Venezuela1.1000.9631.6170.9281.0980.962
Asia
China0.9160.8891.4391.1380.9160.801
Hong Kong SAR1.0891.0610.8020.7831.0811.151
India1.0871.0361.0451.0311.0531.015
Indonesia1.1091.0680.8550.9771.0750.968
Korea1.0270.9650.8840.8551.2931.036
Malaysia1.0401.0620.9180.7151.0951.023
Singapore1.0261.0330.9731.0071.0021.155
Taiwan Province of China1.1090.9690.8630.9771.2461.010
Thailand1.0340.9270.9000.9981.0281.029
Source: Author’s calculations.
Source: Author’s calculations.
Table 23.Comparison of WEO and Consensus Forecasts: Ratios of Root-Mean-Squared Forecast Errors(Consensus over WEO; Consensus measured in April/October)
Real GDPInflationCurrent Account
Current-

Year
Next-

Year
Current-

Year
Next-

Year
Current-

Year
Next-

Year
G-7 countries
Canada0.9090.9670.8641.2180.8890.985
France0.9860.8491.0030.9730.9981.036
Germany1.0710.7961.0600.9171.0181.048
Italy0.8990.8320.8540.8910.8890.828
Japan0.9720.8730.8860.6560.9751.028
United Kingdom1.0640.9400.9890.9961.0101.133
United States0.9370.9580.7890.9221.3510.989
Latin America
Argentina0.8380.9104.1980.7951.0730.874
Brazil1.0010.9370.7620.6890.9350.821
Chile1.0610.8351.2800.7430.9490.956
Colombia1.0631.0131.2800.8671.3770.974
Mexico0.8880.9691.3330.9990.8301.026
Peru1.2180.8931.3451.6560.9620.970
Venezuela0.9510.8851.1530.9530.8780.965
Asia
China0.9080.8631.6181.2990.8260.795
Hong Kong SAR0.9701.0610.9770.9140.9141.055
India1.0711.0491.2820.9740.9380.927
Indonesia0.8520.9611.9971.0100.9050.804
Korea0.8760.9780.9790.9501.0041.002
Malaysia0.8830.9670.7570.6830.9941.005
Singapore0.9301.0341.1151.0520.8881.177
Taiwan Province of China1.0670.9820.9001.0821.1220.944
Thailand0.8960.8590.9221.1980.8220.946
Source: Author’s calculations.
Source: Author’s calculations.

The WEO current-year inflation forecasts are most precise when measured against the February Consensus forecasts for the G-7 countries and Latin America. Surprisingly, however, for seven out of nine Asian economies, the current year February Consensus inflation forecasts are better than the WEO forecasts despite their informational disadvantage. Furthermore, the next-year WEO inflation forecasts do not measure up well against the February Consensus forecasts. Finally the WEO forecasts of the current account generally perform well against the February Consensus forecasts of this variable.

Turning to Table 23 which shows the performance of the April/October Consensus forecasts against the WEO forecasts, it is clear that the Consensus forecasts dominate in the vast majority of cases, the only exception being the current-year forecasts of inflation in Latin America.

Our second exercise makes use of the WEO Board forecasts reported in February and July each year, respectively. Here we report two sets of RMSFE ratios. For the current-year WEO forecasts we report the ratio of the published (April) RMSFE-value divided by the Board (February) RMSFE value, while for the next-year WEO forecasts we report the ratio of the published (September) RMSFE ratio over the July Board RMSFE value. This gets compared to the similar ratios for the Consensus forecasts computed as the ratio of the current year March over the February RMSFE values and the next-year September over the July RMSFE values. As in the earlier analysis, values below unity indicate an improvement in the forecast precision and the amount by which a ration lies below unity is a measure of the degree of the improvement in the precision.

Table 24 shows the outcome of this exercise. The analysis is confined to GDP data since we only have Board data on this variable. Furthermore, we only report results for the G-7 economies and Asia since we do not have complete July Consensus forecast data on the Latin American economies. With few exceptions (most occurring for India where the WEO and Consensus forecasts are not directly comparable due to a difference in timing) the RMSFE ratios fall below unity and often by some distance. Comparing the WEO and Consensus ratios for the current-year G-7 GDP forecasts, in six of seven cases the ratio is lowest for the WEO forecasts. Bearing in mind that the timing of the WEO and Consensus forecasts is not perfect, this suggests that the WEO revisions between the Board and Publication date are relatively efficient. For the next year G-7 GDP forecasts the two sets of RMSFE ratios are quite similar as they are for the current- and next-year forecasts of GDP growth in Asia.

Table 24.WEO Forecasts: RMSFE Ratios for Updates of Real GDP Projections(Published forecasts over forecasts presented to the IMF’s Executive Board)
April Current-Year ForecastsSeptember Next-Year Forecasts
WEOConsensusWEOConsensus
G-7 countries
Canada0.5810.7220.9340.991
France0.6510.7260.9960.837
Germany0.6970.8260.9080.863
Italy0.8370.9150.8820.865
Japan0.8140.8720.8080.885
United Kingdom0.5650.8420.8260.901
United States0.8410.7851.0160.880
Asia
China0.9180.9980.8720.888
Hong Kong SAR0.8540.8240.9821.061
India1.0311.1251.0051.021
Indonesia0.4920.7360.8860.908
Korea0.8170.8711.0121.027
Malaysia0.9380.8200.9200.873
Singapore0.9250.9151.0000.977
Taiwan Province of China0.9670.9920.9990.992
Thailand0.9360.8501.0000.801
Source: Author’s calculations.
Source: Author’s calculations.

XII. Forecast Combinations

Emanating from the classic study by Bates and Granger (1969), a long literature on forecast combinations summarized by, inter alia, Clemen (1989), Diebold and Lopez (1996), and Timmermann (2005), has found evidence that combined forecasts tend to produce better out-of-sample performance than individual forecasting models. For example, Stock and Watson (2001) reported broad support for a simple combination of forecasts in a study of a large cross-section of macroeconomic and financial variables.

Basic intuition for this finding comes from viewing the underlying forecasts as assets in a larger portfolio of forecasts. As long as the forecast errors are not perfectly correlated, combination offers a way to diversify across forecast errors and hence reduce the variance and the root mean squared forecast error. To see this, consider combining two risky assets to form a portfolio. Let ω and (1 − ω) be the weights on the two assets and assume that their returns have variances of σ12 and σ22 and a covariance of σ12. Then the portfolio variance is

and the variance-minimizing weight on the first asset, ω, is simply

By analogy, consider combining the Consensus and WEO forecasts to minimize the MSFE. If the Consensus forecast does not add any value to the WEO forecast, then it should get a weight of zero in such a combination. Suppose that the underlying forecast errors have zero mean with variance σWEO2, σcons2 and covariance σcons,WEO. Then the population values of the combination weight on the WEO and Consensus forecasts that minimize the MSFE are given by

Notice that the weight on the WEO forecast is larger, the greater the variance of the Consensus forecast error, σcons2. Furthermore, negative weights are possible whenever the covariance term, σcons,WEO exceeds one of the variances. In this case one might want to take a “short” position in one of the forecasts. Although it involves taking a contrarian view, this situation is most likely to happen when the two forecasts are highly collinear so the correlation between econs and eWEO is close to one. As we shall see, this is not just a theoretical curiosity, but is fairly common empirically.

Another reason for the potentially superior performance of combinations is that simple averages of forecasts may be more robust to structural breaks and model breakdown which could affect individual forecasting models more strongly as some models learn faster than others and adapt more rapidly to policy or “regime” shifts, c.f. Aiolfi and Timmermann (2005).

Estimates of the combination weights can be obtained from the following current year and next year regressions (c.f. Granger and Ramanathan, 1984):

An intercept is included to allow for the possibility that the underlying forecasts are biased, so our earlier assumption that the WEO and Consensus forecasts were unbiased does not really matter. As we have seen, the assumption of a zero bias cannot always be maintained. For the same reason we do not impose the restriction that λ1 + λ2 = 1, which would make sense if it were known that both sets of forecasts are unbiased. Estimates of λ1 and λ2 show how much weight an outsider should put on the WEO and Consensus forecasts, respectively. Estimates of λ1 close to or above unity suggest that the WEO forecast contains significant information relative to the information in the Consensus forecast. However, even in such cases, the Consensus forecast can be very helpful for a forecaster. For example values of λ1 = 2 and λ2 = −1 would suggest that both the WEO forecast and the spread between the WEO and Consensus forecast help predict the variable of interest.

Table 25 reports results from the forecast combination regressions. Both current and next year GDP forecasts for the G-7, Latin American, and Asian economies generate large and positive weights on the WEO forecasts of many economies, particularly among the G-7 and Latin American countries. Important exceptions include Italy, Germany, the United States, Indonesia, and Thailand which see a negative weight on the WEO forecasts. In contrast, more than half of the weights on the Consensus forecasts are negative. Interestingly, many of the coefficients on the current year WEO forecasts for the G-7 economies and Latin America are significant.

Table 25.Comparison of WEO and Consensus Forecasts: Weights in Granger-Ramanathan Forecast Combination Regressions
WEOConsensus
Estimatet-StatisticsEstimatet-Statistics
Real GDP
G-7 countries
Current-year
Canada0.4470.3240.6190.435
France1.7341.969-0.865