Metaresearch for Evaluating Reproducibility in Ecology and Evolution

Abstract

Recent replication projects in other disciplines have uncovered disturbingly low levels of reproducibility, suggesting that those research literatures may contain unverifiable claims. The conditions contributing to irreproducibility in other disciplines are also present in ecology. These include a large discrepancy between the proportion of “positive” or “significant” results and the average statistical power of empirical research, incomplete reporting of sampling stopping rules and results, journal policies that discourage replication studies, and a prevailing publish-or-perish research culture that encourages questionable research practices. We argue that these conditions constitute sufficient reason to systematically evaluate the reproducibility of the evidence base in ecology and evolution. In some cases, the direct replication of ecological research is difficult because of strong temporal and spatial dependencies, so here, we propose metaresearch projects that will provide proxy measures of reproducibility.

Open science, reproducibility, and transparency in ecology

“…, Leek and Peng , Fidler et al. ). One practice is for authors to share both code and data that generated the specific results shown in their peer‐reviewed publications (Poisot et al.…”

Accumulating evidence in ecology: Once is not enough

“…In the field of psychology, replication of 100 studies found that replication effects were half the size of original study effects, and only 39% of these effects were judged to have replicated original results (Open Science Collaboration, ). There is growing recognition that reproducibility is likely to be an important problem in ecology as well (Ellison, ; Fidler et al, ; Ives, ; Parker et al, ; Schnitzer & Carson, ). The lead entry in a list of proposals developed by Begley and Ioannidis () to deal with the reproducibility crisis and improve quality of scientific research was for editors to solicit replication bids, rewarding investigators willing to undertake serious efforts at replicating published results (Wagenmakers & Forstmann, ).…”

“…The biomedical community has devoted substantial attention to this issue, developing several promising approaches for dealing with different sources of study bias, for example (Turner, Spiegelhalter, Smith, & Thompson, ). Nonetheless, the following claim by Ioannidis () is sobering: “Few systematic reviews and meta‐analyses are both non‐misleading and useful.” Recent calls for transparency in reporting ecological research have been directed largely at increasing the utility of published investigations for research summaries and meta‐analyses (Ellison, ; Fidler et al, ; Parker et al, ; Schnitzer & Carson, ). Importantly from the perspective of this essay, the opportunistic nature of most meta‐analyses precludes design of component studies and does not provide natural opportunities for designing sequences of studies.…”

A Proposal to Advance Theory and Promote Collaboration in Tropical Biology by Supporting Replications

“…The response of the EVE community to these calls has ranged 18 from tepid to skeptical (e.g., Ecology Letters Senior Editors 2016, Schnitzer & Carson 2016) for 19 reasons that echo those put forward by scientists in other fields: a lack of incentives or 20 professional rewards for carrying out replications, few journals willing to publish the results of 21 replicated studies, and concerns about efficient use of scarce research funding (Nakagawa & 22 Parker 2015, Schnitzer & Carson 2016, Fidler et al 2017. In addition to these practical 23 obstacles, however, many suggest a more fundamental conceptual obstacle --that research in 24 EVE is inherently impossible to replicate because it is carried out under unique biotic and abiotic 25 conditions (e.g., species composition and abundance, weather conditions, site history).…”