Systematic Review of the use of “Magnitude-Based Inference” in Sports Science and Medicine

Abstract

Magnitude-based inference (MBI) is a controversial statistical method that has been used in hundreds of papers in sports science despite criticism from statisticians. To better understand how this method has been applied in practice, we systematically reviewed 232 papers that used MBI. We extracted data on study design, sample size, and choice of MBI settings and parameters. Median sample size was 10 per group (interquartile range, IQR: 8 – 15) for multi-group studies and 14 (IQR: 10 – 24) for single-group studies; few studies reported a priori sample size calculations (15%). Authors predominantly applied MBI’s default settings and chose “mechanistic/non-clinical” rather than “clinical” MBI even when testing clinical interventions (only 14 studies out of 232 used clinical MBI). Using these data, we can estimate the Type I error rates for the typical MBI study. Authors frequently made dichotomous claims about effects based on the MBI criterion of a “likely” effect and sometimes based on the MBI criterion of a “possible” effect. When the sample size is n=8 to 15 per group, these inferences have Type I error rates of 12%-22% and 22%-45%, respectively. High Type I error rates were compounded by multiple testing: Authors reported results from a median of 30 tests related to outcomes; and few studies specified a primary outcome (14%). We conclude that MBI has promoted small studies, promulgated a “black box” approach to statistics, and led to numerous papers where the conclusions are not supported by the data. Amidst debates over the role of p-values and significance testing in science, MBI also provides an important natural experiment: we find no evidence that moving researchers away from p-values or null hypothesis significance testing makes them less prone to dichotomization or over-interpretation of findings.

Ingestion of Carbohydrate Prior to and during Maximal, Sprint Interval Cycling Has No Ergogenic Effect: A Randomized, Double-Blind, Placebo Controlled, Crossover Study

“…However, it is worth noting that these conclusions were based on inferences made using magnitude based inferences (MBI), with ‘most likely positive’ and ‘likely positive’ effects of CHO supplementation reported for peak power and tiredness, respectively. The MBI approach has recently received substantial scrutiny for failing to adequately control type 1 error rates (i.e., false positives), particularly for effects characterised as ‘likely’ [ 33 ]. Furthermore, the study of Pomportes et al [ 18 ] also reported (alongside MBI) no significant effect on either variable when assessed using a repeated measures ANOVA.…”

Interdisciplinarity Can Aid the Spread of Better Methods Between Scientific Communities

“…But the paper depends on a statistical method called measurement based inference (MBI), which statisticians have widely criticized as misleading and incoherent, producing high rates of false positive findings (Sainani, 2018). A re-examination of the data from the foam rolling study using more rigorous statistical testing casts doubts on many of their claims (Lohse et al, 2020). Over the years, hundreds of published papers in the discipline of sports science have used MBI, and the unsoundness of the method throws many of these results into doubt (Lohse et al, 2020).…”

“…A re-examination of the data from the foam rolling study using more rigorous statistical testing casts doubts on many of their claims (Lohse et al, 2020). Over the years, hundreds of published papers in the discipline of sports science have used MBI, and the unsoundness of the method throws many of these results into doubt (Lohse et al, 2020). The method was introduced in a 2006 paper by the sports scientists Alan Batterham and Will Hopkins (Batterham and Hopkins, 2006) with the intent of supplanting standard frequentist statistical practices.…”

“…More recently another statistician, Kristin Sainani, pointed out that several claims by the original authors about MBI were incoherent. She also showed how the false positive rate of the method is unacceptably high, especially for studies with small samples sizes (exactly the sort of studies in which it is typically used) (Sainani, 2018;Lohse et al, 2020). Other statisticians have expressed surprise and skepticism upon learning about MBI.…”

Call to increase statistical collaboration in sports science, sport and exercise medicine and sports physiotherapy

“… 54 The problem may be compounded when poor statistical techniques are passed from mentors to students, thus propagating poor practices to the next generation of researchers. 55 Finally, a lack of statistical expertise in the journal peer review process means that many papers are published using suboptimal statistical methods; 52 this often creates a positive feedback loop as methods are copied from paper to paper. BJSM (British Journal of Sports Medicine) only began having statistical Deputy Editors in 2019.…”