The intrinsic predictability of ecological time series and its potential to guide forecasting

Abstract

Successfully predicting the future states of systems that are complex, stochastic and potentially chaotic is a major challenge. Model forecasting error (FE) is the usual measure of success; however model predictions provide no insights into the potential for improvement. In short, the realized predictability of a specific model is uninformative about whether the system is inherently predictable or whether the chosen model is a poor match for the system and our observations thereof. Ideally, model proficiency would be judged with respect to the systems' intrinsic predictability -the highest achievable predictability given the degree to which system dynamics are the result of deterministic v. stochastic processes. Intrinsic predictability may be quantified with permutation entropy (PE), a model-free, information-theoretic measure of the complexity of a time series. By means of simulations we show that a correlation exists between estimated PE and FE and show how stochasticity, process error, and chaotic dynamics affect the relationship. This relationship is verified for a dataset of 461 empirical ecological time series. We show how deviations from the expected PE-FE relationship are related to covariates of data quality and the nonlinearity of ecological dynamics.These results demonstrate a theoretically-grounded basis for a model-free evaluation of a system's intrinsic predictability. Identifying the gap between the intrinsic and realized predictability of time series will enable researchers to understand whether forecasting proficiency is limited by the quality and quantity of their data or the ability of the chosen forecasting model to explain the data. Intrinsic predictability also provides a model-free baseline of forecasting proficiency against which modeling efforts can be evaluated. GlossaryActive information: The amount of information that is available to forecasting models (redundant information minus lost information; Fig. 1). Forecasting error (FE):A measure of the discrepancy between a model's forecasts and the observed dynamics of a system. Common measures of forecast error are root mean squared error and mean absolute error. Entropy: Measures the average amount of information in the outcome of a stochastic process. Information: Any entity that provides answers and resolves uncertainty about a process. When information is calculated using logarithms to the base two (i.e. information in bits), it is the minimum number of yes/no questions required, on average, to determine the identity of the symbol (Jost 2006). The information in an observation consists of information inherited from the past (redundant information), and of new information. Intrinsic predictability: the maximum achievable predictability of a system (Beckage et al. 2011). Lost information:The part of the redundant information lost due to measurement or sampling error, or transformations of the data (Fig. 1). New information, Shannon entropy rate: The Shannon entropy rate quantifies the average amount of information per observation in a t...

Forecasting seed production in perennial plants: identifying challenges and charting a path forward

“…In other words, how predictable is masting, and what is the achievable level of accuracy for the models? It is important to understand if the system is inherently predictable to evaluate the quality of the chosen model (Pennekamp et al, 2019). The impact of chaotic dynamics, the effect of changing initial conditions on model results, should also be considered when assessing the models (Rogers et al, 2022).…”

“…To determine the realistic level of forecast accuracy of the models, it is important to assess the intrinsic predictability of the seed production time series. This includes determining the extent to which the masting system is inherently predictable, and evaluating the chosen model quality in relation to the system's intrinsic predictability (Pennekamp et al, 2019).…”

“…With this opportunity, we aim to develop mast forecasting ("mast cast") as a new branch of the discipline. Masting may offer high potential for ecological forecasting, as the data consists of extended time series with well-established abiotic drivers, potentially providing high intrinsic predictability (Pennekamp et al, 2019;Pearse et al, 2016Pearse et al, , 2021. Additionally, the efforts to develop mast forecasting will lead to wider benefits for the discipline, as it will require the formulation of quantitative predictions and challenge researchers to compare the explanatory power of empirical and process-based models across different geographic regions and ecological contexts (Lewis et al, 2022).…”

On the forecastability of food insecurity

“…Permutation entropy (PE) is a modelfree measure of time series complexity [22,23], that is conceptually similar to the Shannon entropy but is based on the frequency distribution of motifs. PE has been extensively used to assess the predictability of time series in different domains including finance and economics [24,25], ecology [26] and infectious disease epidemiology [12]. In short, to compute the PE of a time series we translate its real valued sequence (x 1 , x 2 , .…”

“…In the Methods section we provide a complete formal definition of the PE and its computation. It has been shown that PE can be considered as a measure of intrinsic predictability of a time series and its value is positively associated with forecasting error [26]. Intuitively, PE quantifies the information that is transmitted from the past to the present state of a time series: a time series that periodically visits the same few symbols among the many possible will have a low entropy and its present state will be easily determined from the past.…”