Appendix I Technical Issues in Estimating Progress in the Millennium Development Goals
- International Monetary Fund
- Published Date:
- May 2011
Measuring Millennium Development Goals performance
The Millennium Development Goals (MDGs) are typically defined in terms of the number or percentage of people (for example, halving the number of poor people or achieving 100 percent access to primary education). Whereas data are generally collected on a country basis, the influence of each country in the global average depends on the size of its population. When large countries like China and India are doing well—as on the poverty MDG—their progress will be reflected very visibly in the global average, but will also hide progress (or a lack of it) in smaller countries. To examine how poor countries are doing, the data in chapter 1 are also presented in terms of progress in individual countries—not to replace the standard approach, but to provide additional information.
MDG performance is measured by deviations from target values required to reach development goals. The reference year for measuring progress is officially set as 1990. Countries are classified as
- on target: the country has already achieved or will meet the 2015 development goal if progress continues until 2015 at the same rate as progress from 1990 to the latest available year;
- off target: the country will not achieve the 2015 development goal if progress continues until 2015 at the same rate as progress from 1990 to the latest available year;
- close to the target or off target and above average: the country is off target, but performing better than the average off-target country; or
- far from the target or off target and below average: the country is off target and performing worse than the average off-target country.
To determine whether a country is on target, we calculate the linear annualized rate of improvement from the 1990 value of each MDG indicator needed to reach the 2015 goal. We restrict our attention to the six MDGs and nine development targets with quantifiable 2015 goals. A country is classified as on target if the observed MDG performance is equal to or above this required achievement path; a country is considered off target if MDG progress is below this path.
Within the off-target category, countries are classified in relation to the group’s average distance to be on target. (The average here is the mean of the off-target group, not the entire sample of countries.) Two subgroups are identified: countries that are off target and above average or close to the target (that is, countries for which development goals are possibly within reach); and countries that are off target and below average or far from the target (that is, countries lagging the most on reaching the 2015 MDGs). Performance data for these two subgroups are presented in tables A1.2, A1.3, and A1.4.
|MDG 3.a||MDG 3.a||MDG 7.c|
|MDG 2.a||gender||gender||MDG 4.a||access|
|Total||MDG 1.a||primary||parity in||parity in||child||MDG 5.a||to safe||MDG 7.c|
|Income level and||number of||extreme||MDG 1.c||education||primary||secondary||mortality||maternal||drinking||access to|
|geographic region||countries||poverty||hunger||completion||education||education||under five||mortality||water||sanitation|
|East Asia and Pacific||24||8||3||15||19||17||24||15||21||21|
|Europe and Central|
|Latin America and|
|Middle East and|
Detailed historical data on MDG performance are required to calculate the achievement path for each country to meet each of the MDGs.1 Unfortunately, such data are not available in many countries for 1990, although estimates for recent years tend to be more complete. If no country data are available for 1990, we used the closest available information in the late 1980s or early 1990s as substitutes for the starting point (table A1.5), and then calculated the rate of progress required from that point to meet the MDG. This approach may be inaccurate if the data for the available starting point are significantly different from the level of MDG performance in 1990 or if the sample period does not capture the latest progress. The latter is a particularly important issue now because data generally are not available for 2009, the crisis year. In addition, for countries without at least two data points, progress cannot be measured, even if data are available for a recent year. Even so, the approach enables us to include more countries than if we relied only on data from 1990 and 2008. Table A1.1 summarizes the number of observations by MDG, income group of countries, and region (World Bank’s classification, table A1.13).
|MDG 2.a||MDG 3.a||MDG 3.a||MDG 4.a||access|
|MDG 1.a||primary||gender parity||gender parity||child||MDG 5.a||to safe||MDG 7.c|
|extreme||MDG 1.c||education||in primary||in secondary||mortality||maternal||drinking||access to|
|Geographic region||poverty||hunger||completion||education||education||under five||mortality||water||sanitation|
|East Asia and Pacific||100||100||100||75||100||33||64||56||62|
|Europe and Central Asia||0||—||100||100||100||92||40||50||27|
|Latin America and the Caribbean||25||100||100||91||100||85||43||60||39|
|Middle East and North Africa||0||50||80||67||33||50||80||44||50|
|MDG 3.a||MDG 3.a||MDG 7.c|
|MDG 2.a||gender||gender||MDG 4.a||access|
|MDG 1.a||primary||parity in||parity in||child||MDG 5.a||to safe||MDG 7.c|
|extreme||MDG 1.c||education||primary||secondary||mortality||maternal||drinking||access to|
|Income level||poverty||hunger||completion||education||education||under five||mortality||water||sanitation|
|Alternative typology||MDG 1.a|
|A. IDA classification|
|B. State fragility|
|Little to low||25||50||100||100||100||53||47||63||47|
|High to extreme||70||67||27||23||38||26||33||64||23|
|C. Export sophistication|
|MDG performance||MDG 1.a|
primary education completion
child mortality under five
|Close to the target||55.97||25.66||68.22||91.97||79.75||87.94||525.39||38.15||51.19|
|Far from the target||9.70||23.04||26.09||73.65||50.92||112.47||485.98||24.52||53.03|
The multinomial logit estimates
Empirically analyzing the MDGs from a cross-country perspective imposes serious challenges, with frequent data gaps and measurement errors. Plausible functional forms are therefore difficult to derive. And several MDGs—such as access to safe drinking water and health targets—are likely to be significantly cross-correlated. Here we discuss some of the most significant issues affecting our methodological approach.2
We employ the multinomial logit estimation method because it is well suited to examine the likelihood that countries fall into one of the three country groups explained above (on target, close to the target, and far from the target), given changes in economic growth and the policy framework. This method is typically employed to model individual discrete choices, such as the occupational choice of households in microsimulations or demand for modes of transportation.
Dependent variable and estimation method. The multinomial logit model does not use the actual values of MDG performance indexes. Instead, MDG performance is defined in terms of three values: 1 for countries far from the target (off target and below average), 2 for countries close to the target (off target and above average), and 3 for countries on target. Avoiding the use of the actual value of MDG indexes is important for two reasons. First, the index numbers that indicate progress in many MDG indicators display substantial variability for countries performing well below or above average. Taking account of this variability would require some form of data trimming, outlier identification procedure, or inclusion of control variables that would reduce the available degrees of freedom and therefore decrease the reliability of estimates, in a context of small data samples. Second and more important, our goal is to assess the likelihood of each country achieving or being on track to achieve the MDGs, conditional on current development performance, an empirical approach consistent with the use of models of categorical dependent variables. We are not trying to determine how much per capita GDP must grow or institutions and policies must improve to attain the development goals by 2015—for which observed values of development indicators and linear regression models are better suited (although these models, as well as nonlinear approaches, may suffer from endogeneity and multicollinearity problems).3
At first glance, an ordinal regression model seems appropriate to analyze the extent to which GDP growth and the policy framework determine the likelihood of being on track to achieve the MDGs. Our initial work therefore involved estimating this relationship using an ordered logistic regression model. However, a fundamental assumption of such models is that the explanatory variables have the same impact across different values for the dependent variable (the proportional odds hypothesis), and this assumption is consistently rejected in most of the nine specifications under consideration. These rejections imply that the coefficients associated with per capita growth and institutions are not equal across levels of MDG performance. For this reason, alternative and less restrictive models that can integrate a differentiated impact of growth and policy on the dependent variable (MDG performance) are required. Consequently, we turn to the multinomial logit model, a nominal outcome estimation technique that reduces the risk of bias resulting from the rejection of the proportional odds hypothesis in the ordinal regression approach, but at the cost of a potential loss of efficiency, given the many parameters in the model.4
Functional form, model specification, and the use of the Country Policy and Institutional Assessment index. The empirical model follows a simple structure that addresses the fundamentals of the Global Monitoring Report framework,5 builds on the current MDG literature, and takes into account data limitations. Progress toward the 2015 goals is expressed as a function of initial conditions and development progress over time (table A1.6). In this context, measuring the quality of policies and institutions represents a major challenge. We use the Country Policy and Institutional Assessment (CPIA) index, which provides a comprehensive assessment of social policies and public sector management that is fundamental for MDG attainment. For example, the CPIA index takes into consideration the extent to which the pattern of public expenditures and revenue collection affects the poor and is consistent with national poverty reduction priorities (criterion 8, equity of public resource use). However, an important limitation of the CPIA index is that values are not strictly comparable across years because of numerous methodological changes over time. Therefore, we focus on current index values, assuming that the current level of the CPIA indexes partially reflects past performances, given the slowly evolving nature of institutions.
The independence of irrelevant alternatives. One assumption of our estimation procedure is that the results satisfy the independence of irrelevant alternatives (IIA) assumption. This assumption, which is often used in the context of public choice theory, is that the odds of an outcome (in this exercise, being on target, close to target, or far from target) do not depend on other alternatives that are available (alternatives are dissimilar). This means that the coefficients on independent variables (in this exercise, growth and the quality of institutions) would not change significantly if we were to suppress one of the categories of the dependent variable and reestimate the model. We performed Small-Hsiao tests6 to determine whether results satisfy the IIA assumption, with generally satisfactory results (calculations of the Small-Hsiao test are shown in table A1.7).7 In any event, it is generally acknowledged that IIA tests have little power in small samples and may even provide conflicting results.8 According to McFad-den, the multinomial logit model “should be limited to situations where the alternatives can plausibly be assumed to be distinct and weighted independently in the eyes of each decision-maker.”9 Therefore, the validity of our conclusions (in terms of the IIA assumption) relies more on the fact that our categories are conceptually independent than on this econometric test.
Endogeneity, reverse causality. Indicators of progress in human development (our dependent variables) can have an impact on growth and the quality of institutions (our independent variables). Thus, our estimations could be subject to reverse causality, which would mean that the coefficients on the independent variables are not correctly estimated. However, such concerns are likely to be less of a problem in our estimation than in regressions using the levels of MDGs (for example, where the level of the poverty head count is the dependent variable). This is because small changes in the dependent variable (for example, poverty head count) in the latter case may have a direct impact on the independent variable (for example, growth). In our estimations, the dependent variable is inclusion in a group defined by deviations from an exogenously determined path (for example, the rate of change in poverty necessary to achieve the goal). The connection between inclusion in one of the three groups and growth is much more tenuous.
|MDG 1.a||MDG 1.c||MDG 2.a||MDG 3.a (primary)||MDG 3.a (secondary)||MDG 4.a||MDG 5.a||MDG 7.a (water)||MDG 7.a (sanitation)|
|(2005 Id PPP)|
|capita 1990||-0.010||0.041*||0.001||0.002||0.079**||0.087**||0.008||0.005||0.146**||0.141 **||-0.005||-0.001||-0.025**||0.002||0.051***||0.046***||-0.005||-0.012|
|(2005 Id PPP)|
|(i)||Close to the target||-9.65||-8.21||2.88||6.0||0.82|
|(2)||Close to the target||-0.01||0.00||0.03||6.0||1.00|
|(3)||Close to the target||-0.03||0.00||0.05||6.0||1.00|
|(4)||Close to the target||-2i.i3||-11.46||19.33||6.0||0.00|
|(5)||Close to the target||-0.01||0.00||0.02||6.0||1.00|
|(6)||Close to the target||-13.66||-12.33||2.65||6.0||0.85|
|(7)||Close to the target||-14.32||-11.65||5.34||6.0||0.50|
|(8)||Close to the target||-12.54||-9.20||6.67||6.0||0.35|
|(9)||Close to the target||-24.05||-19.67||8.76||6.0||0.19|
Predicted probabilities, marginal effects, and odds ratios. Table A1.8 summarizes the effect of marginal changes in independent variables on the probability of a country being in one of our three groups at average sample values. Results show that a one-unit marginal increase in per capita GDP growth is significantly and inversely related to the probability of a country being far from the target in all MDGs, excluding completion of primary schooling and gender parity in secondary education. Conversely, a one-unit increase in GDP per capita growth significantly raises the probability of a country being on target by at least 0.05, holding other variables at their mean, for primary completion, gender parity in secondary education, and access to safe water and sanitation. In addition, CPIA scores appear to have significant marginal effects, at average values, on the probabilities of being on target (positive signs) and/or far from target (negative signs) for several health-related MDGs (hunger, child mortality, and maternal mortality). Note that for several development goals, the predicted probability of a country being close to target is significantly and inversely related to changes in per capita growth and the CPIA index (that is, higher growth may reduce the probability of being close to the target). This does not imply that high growth is correlated with poor performance. Rather, countries with relatively high growth may be on track to meet the goals, instead of off-track but close to the target.
The odds ratios or factor change coefficients (table A1.9) illustrate the dynamics among MDG performance outcomes. These coefficients depict the expected change in the probability of a country being on target versus far from the target and on target versus close to the target, following a one-standard-deviation increase in development drivers and holding all other variables constant. (Results are discussed in chapter 1.)
|MDG 3.a||MDG 3.a||MDG 7.a||MDG 7.a|
|MDG l.a||MDG l.c||MDG 2.a||(primary)||(secondary)||MDG 4.a||MDG 5.a||(water)||(sanitation)|
|Due to increase in GDP per capita growth||Due to increase in CPIA index|
|MDG||on target vs.|
far from target
|on target vs.|
close to target
|on target vs.|
far from target
|on target vs.|
close to target
|MDG l.a extreme poverty||93||19||l2||-19|
|MDG l.c hunger||88||15||281||128|
|MDG 2.a primary completion rate||1,111||180||7||16|
|MDG 3.a gender parity (primary)||l4l||-25||67||12|
|MDG 3.a gender parity (secondary)||1,191||143||-34||16|
|MDG 4.a child mortality under five||163||68||152||36|
|MDG 5.a maternal mortality||189||101||120||34|
|MDG 7.c access to safe water||61||32||-48||58|
|MDG 7.c access to sanitation||102||25||26||-15|
Disaggregation of the CPIA index. Our main results use the aggregate CPIA index as the independent variable. However, we also did estimations using the four components of the CPIA index in 2009 (economic management, structural policies, policies for social inclusion and equity, and public sector management and institutions) separately. These results show that policies for social inclusion and equity (gender equality, equity of public resource use, building human resources, social protection, and labor) are significantly and positively correlated with the odds of a country being on target to achieve the MDGs for extreme poverty, primary completion, gender parity, and child mortality. Results are mixed for the three remaining indicators (secondary education, access to water and sanitation, and maternal mortality) and vary according to whether a country is or is not close to the target. However, these results have to be interpreted with caution because of the high collinearity between CPIA sub-categories (pairwise correlations above 0.65) and the limited degrees of freedom, given the substantial number of parameters and the relatively small size of available data samples.
Alternative measures of policy and institutions. Given the uncertainties surrounding measurements of policy and institutional performance, we test the robustness of our results by including in the analysis other indicators of government performance: state fragility, as measured by Marshall and Cole’s index10 and governance indicators produced by Kaufmann, Kraay, and Mastruzzi.11 The impact of these institutional variables, as well as the CPIA index, on the odds of being on target is summarized in tables A1.10 and A1.11.
table A1.10 shows the links between fragility and governance variables and the odds of a country being on target or far from the target. Results are broadly consistent with our previous estimates using the CPIA index: perceptions of political stability and regulatory quality are positively related to the likelihood of achieving the hunger target; perceptions of government effectiveness, regulatory quality, rule of law, and control of corruption are positively related to achieving the targets for gender equality and primary education; perceptions of state fragility are inversely related to child mortality; and perceptions of regulatory quality are positively correlated with achieving the target for maternal mortality.
Table table A1.11 displays the relationship between institutional indicators and the probability of being on target versus close to the target. Previous results are partly confirmed. Political stability and government effectiveness are positively linked to the hunger target. And political stability, regulatory quality, and control of corruption exhibit positive correlations with access to safe drinking water.
|MDG 3.a||MDG 3.a||MDG 7.c||MDG 7.c|
|Indicator||MDG 1.a||MDG 1.c||MDG 2.a||(primary)||(secondary)||MDG 4.a||MDG 5.a||(water)||(sanitation)|
|CPIA index (2009)||+||+||+||+|
|Rule of law||+||-|
|MDG 3.a||MDG 3.a||MDG 7.c||MDG 7.c|
|Indicator||MDG 1.a||MDG 1.c||MDG 2.a (primary)||(secondary)||MDG 4.a||MDG 5.a||(water)||(sanitation)|
|CPIA index (2009)||+||+||+|
|Rule of law|
These estimations provide additional and interesting links between indicators of institutional quality and progress toward the MDGs. For instance, political stability, regulatory quality, and government effectiveness are positively correlated with poverty reduction for countries on target versus those off target and below average (table A1.10); however, per capita GDP growth loses its significance in the case of political stability and regulatory quality (results not shown). Conversely, voice and accountability—that is, civil and political rights—are not significantly related to most MDGs (tables A1.10 and A1.11); and when a significant correlation is found (table A1.11, extreme poverty), the observed sign is negative (the result is counterintuitive). In addition, a negative relationship is found between some other governance variables (particularly, political stability and government effectiveness) and the attainment of several MDGs (child and maternal mortality and access to water and sanitation [tables A1.10 and A1.11]). A complete analysis of the role of governance in achieving the MDGs is beyond the scope of this appendix. However, these apparently counterintuitive outcomes are consistent with the fact that many of the poorest countries are making important progress toward achieving the MDGs, thanks to sustained growth and despite significant institutional weaknesses—a finding that highlights the necessity of a better understanding of the mechanisms through which policies and institutions promote development.
Education outcomes versus outputs. Our results show that policies and institutions are significantly and positively correlated with several health-related MDGs, whereas correlations with education attainment are not as strong. One possibility is that these differences arise from the fact that the health targets are defined in terms of outcomes (for example, child and maternal mortality), whereas the education goals are defined in terms of access (for example, primary education coverage). In part, the selection of MDG indicators reflected the availability of data. In what follows, we try to assess how important institutions are to achieving higher levels of literacy—a broadly available outcome-based measure of educational attainment.
An explicit target for literacy was not included in the MDGs, so defining outcomes in terms of our three categories (on track, off track but above average, and off track and below average) would be difficult. Instead, we calculate the growth rate of the literacy rate between 1990 and 2009 (or the closest available year) and divide the countries into three categories:
- countries below percentile (33), thus exhibiting the slowest progress;
- countries above percentile (33) and below percentile (66), thus close to the median; and
- countries above percentile (66), the best performers.
To provide a rough comparison with access-based indicators, we perform a similar procedure on the primary completion rate. Several multinomial logit models, reproducing our core specification, are then estimated linking education outcomes and access to various proxies of institutional quality, using as the reference category the group of countries below percentile (33).
The results are mixed and, to some extent, counterintuitive (see table A1.12) so it is impossible to draw strong conclusions. However, it is interesting that the CPIA is not significantly related to the outcome indicator (literacy) but in this specification (unlike our earlier results) is significantly related to the access indicator (primary completion rate). Although these results are difficult to interpret, it does not appear that the differing results for the impact of growth and institutions on progress toward achieving the health and education MDGs can be ascribed simply to the use of outcome versus access indicators.
|rate, total||Literacy rate, adult total|
|(% of relevant||(% of people aged 15|
|Indicator||age group)||and above)|
|CPIA index (2009)||+|
|Voice and accountability|
|Rule of law||+|
|Control of corruption||+||+|FIGURE A1.1Odds of achieving the MDGs improve with growth and better policy
Source: World Bank staff calculations.
Note: CRIA = Country Policy and Institutional Assessment. Pairwise correlations are significant at the 0.10 level or better.
|East Asia and Pacific|
|Korea, Dem. Rep.||LIC|
|Micronesia, Fed. Sts.||LMC|
|Papua New Guinea||LMC|
|Europe and Central Asia|
|Bosnia and Herzegovina||UMC|
|Latin America and the Caribbean|
|Antigua and Barbuda||UMC|
|St. Kitts and Nevis||UMC|
|St. Vincent and|
|Venezuela, R.B. de||UMC|
|Middle East and North Africa|
|Egypt, Arab Rep.||LMC|
|ran, Islamic Rep.||UMC|
|Syrian Arab Republic||LMC|
|West Bank and Gaza||LMC|
|Central African Republic||LIC|
|Congo, Dem. Rep.||LIC|
|Säo Tomé and Principe||LMC|
|High-income OECD economies|
|Other high-income economies|
|Cyp r u s|
|Hong Kong SAR, China|
|Isle of Man|
|Macao SAR, China|
|Northern Mariana Islands|
|Trinidad and Tobago|
|Turks and Caicos Islands|
|United Arab Emirates|
|Virgin Islands (U.S.;|
GoD. andA.Quijada.Forthcoming.“Assessing the Odds of Achieving the MDGs.”Background paper for Global Monitoring Report2011World BankWashington, DC.
KaufmannDA.Kraay andM.Mastruzzi.2010. “The Worldwide Governance Indicators: Methodology and Analytical Issues.”Policy Research Working Paper 5430World BankWashington, DC.
LallS.2000. “The Technological Structure and Performance of Developing Country Manufactured Exports, 1985–1998.”Working Paper 44Queen Elizabeth House, University of OxfordOxford, UK.
LofgrenH. andI.Rodarte.Forthcoming.“Macro Analysis of Health and Education MDGs: Brief Review and Country–Level Diagnosis.”Background note for Global Monitoring Report2011 World BankWashington, DC.
LongJ. S. andJ.Freese.2006. Regression Modelsfor Categorical Dependent Variables UsingStata. 2nd edition. College StationTX: Stata Press.
MarshallM. andB.Cole.2010. “Global Report 2009: Confl ict, Governance, and State Fragility.”Center for Global PolicyWashington, DC.
McFaddenD.1973. “Conditional Logit Analysis of Qualitative Choice Behavior.” In Frontiersof Econometricsed. P.Zarembka105–42. New York: Academic Press.
SmallK. andC.Hsiao.1985. “Multinomial Logit Specifi cation Tests.”International EconomicReview26 (3): 619–27.