Predictive models aren't for causal inference

Arif, Suchinta; MacNeil, Aaron

doi:10.1111/ele.14033

Cited by 47 publications

(50 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As single‐season occupancy models are in essence a form of logistic regression, our results have wider implications for the use of information‐theoretic model selection in ecology. In particular, we argue that our results, alongside those of others (Arif & MacNeil, 2022; Luque‐Fernandez et al, 2019; McElreath, 2021), underscore the risks associated with using the information‐theoretic approach to compare biological hypotheses in observational studies. Causal inference and the information‐theoretic approach share similar philosophical underpinnings, and should be seen as complementary tools that accomplish different tasks.…”

Section: Discussionsupporting

confidence: 68%

“…An alternative approach to model selection that has gained recent traction in ecology and evolution (e.g., Arif & MacNeil, 2022; Laubach et al, 2021) is causal inference. Causal inference is concerned with predicting the consequences of intervening in a system, as well as inferring counterfactual outcomes, events that might have happened, under hypothetical unrealized conditions (Pearl et al, 2016, p. 89).…”

Section: Introductionmentioning

confidence: 99%

“…As AIC and other information criteria select models based on their expected predictive performance, they are vulnerable to collider bias: including a collider covariate tends to improve a model's AIC score, while simultaneously resulting in an estimated effect that is far from the true value (Figure 2A–C; Luque‐Fernandez et al, 2019). Consequently, recent studies have argued that it is essential to consider whether the purpose of a model is inference (i.e., explanation) or prediction when deciding on a model selection strategy (e.g., Arif & MacNeil, 2022; Laubach et al, 2021). However, the implications for models like occupancy models, which contain multiple submodels, are unclear.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Model selection in occupancy models: Inference versus prediction

Stewart

Stephens

Hill

et al. 2023

Ecology

View full text Add to dashboard Cite

Occupancy models are a vital tool for ecologists studying the patterns and drivers of species occurrence, but their use often involves selecting among models with different sets of occupancy and detection covariates. The information-theoretic approach, which employs information criteria such as Akaike's information criterion (AIC) is arguably the most popular approach for model selection in ecology and is often used for selecting occupancy models. However, the information-theoretic approach risks selecting models that produce inaccurate parameter estimates due to a phenomenon called collider bias, a type of confounding that can arise when adding explanatory variables to a model. Using simulations, we investigated the consequences of collider bias (using an illustrative example called M-bias) in the occupancy and detection processes of an occupancy model, and explored the implications for model selection using AIC and a common alternative, the Schwarz criterion (or Bayesian information criterion, BIC). We found that when M-bias was present in the occupancy process, AIC and BIC selected models that inaccurately estimated the effect of the focal occupancy covariate, while simultaneously producing more accurate predictions of the site-level occupancy probability than other models in the candidate set. In contrast, M-bias in the detection process did not impact the focal estimate; all models made accurate inferences, while the site-level predictions of the AIC/BIC-best model were slightly more accurate. Our results show that information criteria can be used to select occupancy covariates if the sole purpose of the model is prediction, but must be treated with more caution if the purpose is to understand how environmental variables affect occupancy. By contrast, detection covariates can usually be selected using information criteria regardless of the model's purpose. These findings illustrate the importance of distinguishing between the tasks of parameter inference and prediction in ecological modeling. Furthermore, our results underline concerns about the use of information criteria to compare different biological hypotheses in observational studies.

show abstract

Section: Discussionsupporting

confidence: 68%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Model selection in occupancy models: Inference versus prediction

Stewart

Stephens

Hill

et al. 2023

Ecology

View full text Add to dashboard Cite

show abstract

“…Despite their differing objectives, model selection for the goals of exploration, hypothesis testing, and prediction can all be performed on the basis of predictive assessment. This may seem contradictory, especially for hypothesis testing, since it is known that the model closest to “truth” is not necessarily the best predictive model (Arif & Aaron MacNeil, 2022; Shmueli, 2010) (see Box 3 for further discussion). However, the specified modeling goal strongly constrains which models are included in the candidate set, and for this reason, the interpretation of the selected model will differ even if the same method is used to compare models across the different goals.…”

Section: Predictive Assessment Modeling Goals and The Purpose Of Mode...mentioning

confidence: 99%

Cross validation for model selection: A review with examples from ecology

Yates

Aandahl

Richards

et al. 2023

Ecological Monographs

View full text Add to dashboard Cite

Specifying, assessing, and selecting among candidate statistical models is fundamental to ecological research. Commonly used approaches to model selection are based on predictive scores and include information criteria such as Akaike's information criterion, and cross validation. Based on data splitting, cross validation is particularly versatile because it can be used even when it is not possible to derive a likelihood (e.g., many forms of machine learning) or count parameters precisely (e.g., mixed‐effects models). However, much of the literature on cross validation is technical and spread across statistical journals, making it difficult for ecological analysts to assess and choose among the wide range of options. Here we provide a comprehensive, accessible review that explains important—but often overlooked—technical aspects of cross validation for model selection, such as: bias correction, estimation uncertainty, choice of scores, and selection rules to mitigate overfitting. We synthesize the relevant statistical advances to make recommendations for the choice of cross‐validation technique and we present two ecological case studies to illustrate their application. In most instances, we recommend using exact or approximate leave‐one‐out cross validation to minimize bias, or otherwise k‐fold with bias correction if k < 10. To mitigate overfitting when using cross validation, we recommend calibrated selection via our recently introduced modified one‐standard‐error rule. We advocate for the use of predictive scores in model selection across a range of typical modeling goals, such as exploration, hypothesis testing, and prediction, provided that models are specified in accordance with the stated goal. We also emphasize, as others have done, that inference on parameter estimates is biased if preceded by model selection and instead requires a carefully specified single model or further technical adjustments.

show abstract

“…Despite the simplicity of the approach, there are two shortcomings that are often overlooked. First of all, any variable selection in a regression is for predictive inference (i.e., which model best predicts the response variable) and not for causal inference (i.e., which is the effect of pH on bacterial richness) (see Arif& MacNeil, 2022). In other words, interpreting variable importance from a predictive model is a risky game.…”

Section: Frontiers In Soil Ecologymentioning

confidence: 99%

Frontiers in soil ecology—Insights from the World Biodiversity Forum 2022

Eisenhauer

Bender

Calderón‐Sanou

et al. 2022

J of Sust Agri & Env

View full text Add to dashboard Cite

Global change is affecting soil biodiversity and functioning across all terrestrial ecosystems. Still, much is unknown about how soil biodiversity and function will change in the future in response to simultaneous alterations in climate and land use, as well as other environmental drivers. It is crucial to understand the direct, indirect and interactive effects of global change drivers on soil communities and ecosystems across environmental contexts, not only today but also in the near future. This is particularly relevant for international efforts to tackle climate change like the Paris Agreement, and considering the failure to achieve the 2020 biodiversity targets, especially the target of halting soil degradation. Here, we outline the main frontiers related to soil ecology that were presented and discussed at the thematic sessions of the World Biodiversity Forum 2022 in Davos, Switzerland. We highlight multiple frontiers of knowledge associated with data integration, causal inference, soil biodiversity and function scenarios, critical soil biodiversity facets, underrepresented drivers, global collaboration, knowledge application and transdisciplinarity, as well as policy and public communication. These identified research priorities are not only of immediate interest to the scientific community but may also be considered in research priority programmes and calls for funding.

show abstract

Predictive models aren't for causal inference

Cited by 47 publications

References 53 publications

Model selection in occupancy models: Inference versus prediction

Model selection in occupancy models: Inference versus prediction

Cross validation for model selection: A review with examples from ecology

Frontiers in soil ecology—Insights from the World Biodiversity Forum 2022

Contact Info

Product

Resources

About