Difficulties in benchmarking ecological null models: an assessment of current methods

Abstract

Identifying species interactions and detecting when ecological communities are structured by them is an important problem in ecology and biogeography. Ecologists have developed specialized statistical hypothesis tests to detect patterns indicative of community‐wide processes in their field data. In this respect, null model approaches have proved particularly popular. The freedom allowed in choosing the null model and statistic to construct a hypothesis test leads to a proliferation of possible hypothesis tests from which ecologists can choose to detect these processes. Here, we point out some serious shortcomings of a popular approach to choosing the best hypothesis for the ecological problem at hand that involves benchmarking different hypothesis tests by assessing their performance on artificially constructed data sets. Terminological errors concerning the use of Type I and Type II errors that underlie these approaches are discussed. We argue that the key benchmarking methods proposed in the literature are not a sound guide for selecting null hypothesis tests, and further, that there is no simple way to benchmark null hypothesis tests. Surprisingly, the basic problems identified here do not appear to have been addressed previously, and these methods are still being used to develop and test new null models and summary statistics, from quantifying community structure (e.g., nestedness and modularity) to analyzing ecological networks.

Assembly of the amphibian microbiome is influenced by the effects of land‐use change on environmental reservoirs

“…We acknowledge the limitations associated with ecological null models (Molina and Stone, 2020) as well as our use of a strict taxonomic clustering threshold. First, null models applied to post‐competition pools can underestimate the influence of interspecific competition as the regional pool already reflects the effects of competition (Colwell and Winkler, 1984).…”

Metacommunity dynamics and the detection of species associations in co-occurrence analyses: why patch disturbance matters

“…5c). This underscores once more how inferring species interactions from snap-shots of co-occurrence patterns can be as hazardous as it is appealing, and requires careful thinking about null expectations (Blanchet et al, 2020; Molina and Stone, 2020; Münkemüller et al, 2020). We join the recent burst of cautionary articles in highlighting these concerns, and in any case, we encourage investigators to evaluate whether detected species association patterns could match those reported here, and so could stem from pure metacommunity dynamical effects.…”

“…However, it is well known that several pitfalls can compromise this approach, since correlation is not equivalent to interaction (Blanchet et al, 2020; Molina and Stone, 2020). As a simple example, patch heterogeneity and differential habitat preferences among species can cause species associations, even in the absence of any interaction between the pairs of species involved.…”

A niche for null models in adaptive resource management

“…Most notably, ecological null models are used by community ecologists to expose mechanisms of species interactions involved in species coexistence and patterns of biodiversity (e.g., Adler et al, 2010), and by evolutionary biologists to examine the role of competition in shaping natural selection on trait divergence (e.g., Anderson & Weir, 2021). Though ecological null models are traditionally used in explanatory inference, the same philosophy can be used to increase the rigor of predictive ecological inference (Dietze et al, 2018), and the identification of null models for specific types of data continues to be a topic of inquiry (Molina & Stone, 2020).…”