Predicting species abundances in a grassland biodiversity experiment: Trade‐offs between model complexity and generality

Abstract

Models of natural processes necessarily sacrifice some realism for the sake of tractability. Detailed, parameter‐rich models often provide accurate estimates of system behaviour but can be data‐hungry and difficult to operationalize. Moreover, complexity increases the danger of ‘over‐fitting’, which leads to poor performance when models are applied to novel conditions. This challenge is typically described in terms of a trade‐off between bias and variance (i.e. low accuracy vs. low precision). In studies of ecological communities, this trade‐off often leads to an argument about the level of detail needed to describe interactions among species. Here, we used data from a grassland biodiversity experiment containing nine locally abundant plant species (the Jena ‘dominance experiment’) to parameterize models representing six increasingly complex hypotheses about interactions. For each model, we calculated goodness‐of‐fit across different subsets of the data based on sown species richness levels, and tested how performance changed depending on whether or not the same data were used to parameterize and test the model (i.e. within vs. out‐of‐sample), and whether the range of diversity treatments being predicted fell inside or outside of the range used for parameterization. As expected, goodness‐of‐fit improved as a function of model complexity for all within‐sample tests. In contrast, the best out‐of‐sample performance generally resulted from models of intermediate complexity (i.e. with only two interaction coefficients per species—an intraspecific effect and a single pooled interspecific effect), especially for predictions that fell outside the range of diversity treatments used for parameterization. In accordance with other studies, our results also demonstrate that commonly used selection methods based on AIC of models fitted to the full dataset correspond more closely to within‐sample than out‐of‐sample performance. Synthesis. Our results demonstrate that models which include only general intra and interspecific interaction coefficients can be sufficient for estimating species‐level abundances across a wide range of contexts and may provide better out‐of‐sample performance than do more complex models. These findings serve as a reminder that simpler models may often provide a better trade‐off between bias and variance in ecological systems, particularly when applying models beyond the conditions used to parameterize them.

Ecological forecasts reveal limitations of common model selection methods: predicting changes in beaver colony densities

“…Goodness‐of‐fit is a useful diagnostic, but it cannot be used to select predictive models from a set of candidates since the fit of a model always increases with the addition of covariates (although this may be moderately accounted for by using an adjusted R 2 ). Information criteria (e.g., AIC, BIC, DIC) are statistical tools useful for correcting goodness‐of‐fit by including a term that penalizes increases in model complexity (as measured by the number of parameters and sample size; Burnham and Anderson 2002, Clark et al 2020). The standard model selection framework using information criteria involves fitting competing models to the same data and then assessing these models with information criteria to determine the most parsimonious model.…”

Identifying “Useful” Fitness Models: Balancing the Benefits of Added Complexity with Realistic Data Requirements in Models of Individual Plant Fitness

“…The growing interest in understanding how species interactions within ecological communities modulate coexistence as well as community diversity and stability, has driven a reassessment of the assumptions and simplifications commonly made in individual fitness modeling. Clark et al (2019) found that too much added information can degrade the accuracy of fitness models, but also found that the best models are not the simplest models that are commonly used. Though higher-order interactions are widely recognized to be an important group of interactions in natural communities, the issues they cause in modeling individual fitness are not trivial and have driven the opinion that the value of HOIs is not sufficient to merit the issues they cause with modeling (Levine et al, 2017); a conclusion we feel is premature.…”

“…Further, a recent paper by Clark et al (2019) found that models of intermediate complexity are the best at predicting local patterns of diversity. Clark et al's (2019) careful comparison of models containing different amounts of information made an important point: there is often a trade-off between how much biological information we include in models and their accuracy. It remains unclear, however, which biological details (rather than just how many) are the most important for accurately modeling and predicting coexistence outcomes and patterns of diversity.…”

Structural forecasting of species persistence under changing environments

“…The difficulty in answering this question resides in knowing the exact equations governing the dynamics of ecological systems, together with the high uncertainty regarding the initial conditions, parameter values, intrinsic randomness, and more importantly, how the changing external conditions (such as biotic and abiotic factors) will affect the dynamics (Levins, 1968; Sugihara, 1994; Fukami, 2015; Boettiger, 2018; Cenci and Saavedra, 2018b). This complexity of multidimensional and changing factors has typically taken both theoretical and empirical studies to choose between understanding and predicting species persistence (Petchey et al ., 2015; Clark et al ., 2020).…”

“…Importantly, biodiversity forecasting spans and integrates many model‐driven (parametric) and data‐driven (nonparametric) methodologies, such as uncertainty propagation, statistics, informatics, Bayesian approaches, machine learning, Markov chain approaches, empirical dynamic modelling (Sugihara et al ., 2012; Harfoot et al ., 2014; Cazelles et al ., 2016; Dietze, 2017; Cenci and Saavedra, 2019; Adams et al ., 2020; Maynard et al ., 2020), as well as parameterising complex mechanistic models using either demographic, eco‐physiological or allometric information (Preston, 1962; Pacala et al ., 1996; Dietze, 2017). However, the majority of these methodologies demands extensive amounts of data, their explanatory power has been contested, and their generalisation has not always been validated with experimental work (Dietze, 2017; Clark et al ., 2020).…”