Complexity is costly: a meta‐analysis of parametric and non‐parametric methods for short‐term population forecasting

Abstract

Short‐term forecasts based on time series of counts or survey data are widely used in population biology to provide advice concerning the management, harvest and conservation of natural populations. A common approach to produce these forecasts uses time‐series models, of different types, fit to time series of counts. Similar time‐series models are used in many other disciplines, however relative to the data available in these other disciplines, population data are often unusually short and noisy and models that perform well for data from other disciplines may not be appropriate for population data. In order to study the performance of time‐series forecasting models for natural animal population data, we assembled 2379 time series of vertebrate population indices from actual surveys. Our data were comprised of three vastly different types: highly variable (marine fish productivity), strongly cyclic (adult salmon counts), and small variance but long‐memory (bird and mammal counts). We tested the predictive performance of 49 different forecasting models grouped into three broad classes: autoregressive time‐series models, non‐linear regression‐type models and non‐parametric time‐series models. Low‐dimensional parametric autoregressive models gave the most accurate forecasts across a wide range of taxa; the most accurate model was one that simply treated the most recent observation as the forecast. More complex parametric and non‐parametric models performed worse, except when applied to highly cyclic species. Across taxa, certain life history characteristics were correlated with lower forecast error; specifically, we found that better forecasts were correlated with attributes of slow growing species: large maximum age and size for fishes and high trophic level for birds. Synthesis Evaluating the data support for multiple plausible models has been an integral focus of many ecological analyses. However, the most commonly used tools to quantify support have weighted models’ hindcasting and forecasting abilities. For many applications, predicting the past may be of little interest. Concentrating only on the future predictive performance of time series models, we performed a forecasting competition among many different kinds of statistical models, applying each to many different kinds of vertebrate time series of population abundance. Low‐dimensional (simple) models performed well overall, but more complex models did slightly better when applied to time series of cyclic species (e.g. salmon).

Expected future performance of salmon abundance forecast models with varying complexity

“…They were reasonably flexible yet relatively parsimonious and were not subject to the difficulties associated with the use of environmental variables described above. Several studies have found that forecast models incorporating temporal and autoregressive changes in abundance, productivity, and recruitment perform relatively well for Pacific salmon populations (Noakes et al 1990;Peterman et al 2000;Haeseker et al 2005;Ward et al 2014). An advantage of the autoregressive jack model (Model 8) over the generalized additive jack trend models (Models 4 and 10) is that one does not have to make an a priori decision about the degree of flexibility to allow for in the estimated temporal change in the jack-to-SI ratio.…”

“…Continued interest in improving forecast accuracy has led to increased forecast model complexity, for example by incorporating environmental predictor variables (Deyle et al 2013;Stige et al 2013). However, increased complexity does not necessarily result in better short-term forecasts (Ward et al 2014), and in a management context there is value in simplicity when it is easier for stakeholders to understand a model and intuit its predictions. Hence, the accuracy of complex forecast models should be carefully evaluated relative to more parsimonious models.…”

Quantitative argument for long-term ecological monitoring

“…However, if a model is a poor description of a system, no amount of additional data will im prove predictability (MacNally 2000, DeAngelis & Yurek 2015. For example, fishery stock prediction has been a difficult challenge despite continual model refinements and new data inputs (Ward et al 2014, Schindler & Hilborn 2015. The limitations of current ecosystem models to predict out of sample may arise in part from observational error in the data.…”

On estimating the reliability of ecological forecasts

“…The prediction model is built using the training set, and expected forecast error is evaluated by predicting the test set (e.g., Ward et al, 2014). This method is computationally efficient, however, it may overestimate expected forecast error due to its incomplete usage of the available data (Molinaro et al, 2005).…”