The Shelf Life of Official Sub-National Population Forecasts in England

Abstract

We measure the empirical distribution of the accuracy of projected population in subnational areas of England, developing the concept of 'shelf life': the furthest horizon for which the subsequent best estimate of population is within 10% of the forecast, for at least 80% of areas projected. Since local government reorganisation in 1974, the official statistics agency has projected the population of each local government area in England: for 108 areas in nine forecasts up to the 1993-based, and for over 300 areas in 10 forecasts from the 1996-based to the 2014-based forecasts. By comparing the published forecast (we use this term rather than projection) with the post-census population estimates, the empirical distribution of errors has been described. It is particularly dependent on the forecast horizon and the type of local authority. For 10year forecasts the median absolute percentage error has been 7% for London Boroughs and 3% for Shire Districts. Users of forecasts tend to have in mind a horizon and a required accuracy that is of relevance to their application. A shelf life of 10 years is not sufficient if the user required that accuracy of a forecast 15 years ahead. The relevant effective shelf life deducts the user's horizon. We explore the empirical performance of official sub-national forecasts in this light. A five-year forecast for London Boroughs requiring 10% accuracy is already beyond its effective shelf life by the time it is published. Collaboration between forecasters and users of forecasts can develop information on uncertainty that is useful to planning.

Introduction

“…The interest of population forecast users in forecast uncertainty noted above is similar to the findings by Wilson and Shalley (2019) for Australia. Using data from a small online survey and subsequent focus groups of subnational population forecast users, the authors find that 90% of users who responded were in favour of receiving information on forecast uncertainty.…”

Evaluation of Sub-National Population Projections: a Case Study for London and the Thames Valley

“…As Yamauchi et al (2017) comment, EPIs are highest in the USA, moderate in Australia, lower in England and lowest in Japan. A second set of EPIs is reported for England (Table 8D), from Simpson et al (2018), which are broadly similar to the first set (UKWIR 2015), though the authors consider variation by LAD type more important than variation by size, with higher EPIs found in London Boroughs. These differences between countries are associated with international differences in internal migration intensity (Bell et al 2015(Bell et al , 2018Rees et al 2016c) and the degree to which population change is driven by international migration.…”

“…NRS 2018, Rees et al (2013), Caswell and Gassen (2015) 1F Probabilistic projections Generation of a large set of projections by sampling from error distributions producing probability distributions of future population Wilson and Bell (2007), Wilson (2013), Sevcikova et al (2018), Raymer et al (2012) 1G Error analysis Use of the historical errors from tested comparisons as empirical predictive intervals in projections Smith et al (2001), Shaw (2007), Shaw (2008), Rayer et al (2009), Tayman (2011), Wilson (2012), Smith et al (2013), Simpson et al (2018) 1H Use of projections Advice on how to use evaluation knowledge, Shelf life Keilman (2008), Wilson et al (2018) Wilson (2018), Simpson et al (2018) The second evaluation approach (Table 1B), Controlled Comparison, involves using a fixed set of inputs (populations and components) and assumptions when running a suite of projections which differ in model design for just one component. Wilson and Bell (2004) test out ten different models for projecting internal migration, including the net migration flow model, the multi-regional model, a pool model and a gravity-type model.…”

“…Note: The % population change axis differs between WRZs exponential relationship (virtually constant for larger populations) with size of population. The original studies either report Absolute Percentage Errors (APE) (Smith et al 2001summarised in Tayman 2011, Yamauchi et al 2017 or Empirical Prediction Intervals (UKWIR 2015) or both (Simpson et al 2018). In Table 8, we bring together the APE or EPI estimates for a sample of countries which have been studied.…”

Subnational population forecasts: Do users want to know about uncertainty?

“…But it is at the subnational scale that the need for warnings about forecast uncertainty is much greater because errors are larger. Only a handful of case studies of subnational population forecasts with prediction intervals exist (e.g., Lee, Miller, and Edwards 2003;Rayer, Smith, and Tayman 2009;Rees and Turton 1998;Simpson, Wilson, and Shalley 2018;Wilson 2013;Wisniowski and Raymer 2016). There is a pressing need for more work on the development of methods and software for quantifying the uncertainty of subnational population forecasts.…”