Magnitude Based Inference in Relation to One-sided Hypotheses Testing Procedures

Aisbett, Janet; Lakens, Daniël; Sainani, Kristin L.

doi:10.31236/osf.io/pn9s3

Cited by 18 publications

(23 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, MBI uses the same underlying hypothesis tests as minimal effect testing and equivalence testing/non-inferiority testing [67]. Thus, MBI practitioners could use these procedures instead [68]. Furthermore, sports scientists should understand that-when used correctly-p-values are valid tools of inference [69].…”

Section: Discussionmentioning

confidence: 99%

“…Similarly, the benefit/positive probability is the p-value associated with the null hypothesis of benefit (H o : effect size �δ b ); and subtracting this value from 1 gives the p-value associated with the null hypothesis of no benefit (H o : effect size �δ b ). One may use these p-values to perform equivalence testing and/or minimal effects testing [67,68]. We give a specific example of how this would be done in Table 4 using a study on foam rolling [30].…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Systematic review of the use of “magnitude-based inference” in sports science and medicine

et al. 2020

View full text Add to dashboard Cite

Magnitude-based inference (MBI) is a controversial statistical method that has been used in hundreds of papers in sports science despite criticism from statisticians. To better understand how this method has been applied in practice, we systematically reviewed 232 papers that used MBI. We extracted data on study design, sample size, and choice of MBI settings and parameters. Median sample size was 10 per group (interquartile range, IQR: 8-15) for multi-group studies and 14 (IQR: 10-24) for single-group studies; few studies reported a priori sample size calculations (15%). Authors predominantly applied MBI's default settings and chose "mechanistic/non-clinical" rather than "clinical" MBI even when testing clinical interventions (only 16 studies out of 232 used clinical MBI). Using these data, we can estimate the Type I error rates for the typical MBI study. Authors frequently made dichotomous claims about effects based on the MBI criterion of a "likely" effect and sometimes based on the MBI criterion of a "possible" effect. When the sample size is n = 8 to 15 per group, these inferences have Type I error rates of 12%-22% and 22%-45%, respectively. High Type I error rates were compounded by multiple testing: Authors reported results from a median of 30 tests related to outcomes; and few studies specified a primary outcome (14%). We conclude that MBI has promoted small studies, promulgated a "black box" approach to statistics, and led to numerous papers where the conclusions are not supported by the data. Amidst debates over the role of p-values and significance testing in science, MBI also provides an important natural experiment: we find no evidence that moving researchers away from p-values or null hypothesis significance testing makes them less prone to dichotomization or over-interpretation of findings.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Systematic review of the use of “magnitude-based inference” in sports science and medicine

et al. 2020

View full text Add to dashboard Cite

show abstract

“…gives the p-value associated with the null hypothesis of no benefit (H o : effect size ≤δ b ). One may use these p-values to perform equivalence testing and/or minimal effects testing [67,68]. We give a specific example of how this would be done in Table 4 using a study on foam rolling [30].…”

Section: Discussionmentioning

confidence: 99%

“…For example, MBI uses the same underlying hypothesis tests as minimal effect testing and equivalence testing/non-inferiority testing[67]. Thus, MBI practitioners could use these procedures instead[68]. Furthermore, sports scientists should understand that-when used correctlyp-values are valid tools of inference[69].…”

mentioning

confidence: 99%

Systematic Review of the use of “Magnitude-Based Inference” in Sports Science and Medicine

Lohse¹,

Sainani²,

Taylor³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Magnitude-based inference (MBI) is a controversial statistical method that has been used in hundreds of papers in sports science despite criticism from statisticians. To better understand how this method has been applied in practice, we systematically reviewed 232 papers that used MBI. We extracted data on study design, sample size, and choice of MBI settings and parameters. Median sample size was 10 per group (interquartile range, IQR: 8 – 15) for multi-group studies and 14 (IQR: 10 – 24) for single-group studies; few studies reported a priori sample size calculations (15%). Authors predominantly applied MBI’s default settings and chose “mechanistic/non-clinical” rather than “clinical” MBI even when testing clinical interventions (only 14 studies out of 232 used clinical MBI). Using these data, we can estimate the Type I error rates for the typical MBI study. Authors frequently made dichotomous claims about effects based on the MBI criterion of a “likely” effect and sometimes based on the MBI criterion of a “possible” effect. When the sample size is n=8 to 15 per group, these inferences have Type I error rates of 12%-22% and 22%-45%, respectively. High Type I error rates were compounded by multiple testing: Authors reported results from a median of 30 tests related to outcomes; and few studies specified a primary outcome (14%). We conclude that MBI has promoted small studies, promulgated a “black box” approach to statistics, and led to numerous papers where the conclusions are not supported by the data. Amidst debates over the role of p-values and significance testing in science, MBI also provides an important natural experiment: we find no evidence that moving researchers away from p-values or null hypothesis significance testing makes them less prone to dichotomization or over-interpretation of findings.

show abstract

“…We applied a minimum effect test (MET) [21] to provide a practical, probabilistic interpretation of the difference in HRex between the recovered and strained states. The MET aims to combine the strength of drawing inferences from the data in relation to meaningful effect sizes with a formal statistical foundation grounded in frequentist approaches to inferences [22]. The MET was performed as part of the MIXED procedure in SAS ® Software, using -1% as the threshold for practical importance.…”

Section: The First Part Of Our Analysis Was Performed Inmentioning

confidence: 99%

Monitoring training and recovery responses with heart rate measures during standardized warm-up in elite badminton players

Schneider¹,

Wiewelhove²,

McLaren³

et al. 2020

Preprint

View full text Add to dashboard Cite

(Preprint under review) Purpose: To investigate short-term training and recovery-related effects on heart rate during a standardized submaximal running test.Methods: Ten elite badminton players (7 females and 3 males) were monitored during a 12-week training period in preparation for the World Championships. Exercise heart rate (HRex) and perceived exertion were measured in response to a 5-min submaximal shuttle-run test during the morning session warm-up. This test was repeatedly performed on Mondays after 1–2 days of pronounced recovery (‘recovered’ state; reference condition) and on Fridays following 4 consecutive days of training (‘strained’ state). In addition, the serum concentration of creatine kinase and urea, perceived recovery–stress states, and jump performance were assessed before warm-up.Results: Creatine kinase increased in the strained compared to the recovered state and the perceived recovery–stress ratings decreased and increased, respectively (range of average effects sizes: |d| = 0.93–2.90). The overall HRex was 173 bpm and the observed within-player variability (i.e., standard deviation as a coefficient of variation [CV]) was 1.3% (90% confidence interval: 1.2% to 1.5%). A linear reduction of -1.4% (-3.0% to 0.3%) was observed in HRex over the 12-week observational period. HRex was -1.5% lower (-2.2% to -0.9%) in the strained compared to the recovered state, and the standard deviation (as a CV) representing interindividual variability in this response was 0.7% (-0.6% to 1.2%).Conclusions: Our findings suggest that HRex measured during a standardized warm-up is sensitive to short-term changes in training load, with HRex decreasing on average in response to intensified training during preparatory training microcycles. From a practical perspective, it seems advisable to determine intra-individual recovery–strain responses by repeated testing, as HRex responses may vary substantially between and within players.

show abstract

Magnitude Based Inference in Relation to One-sided Hypotheses Testing Procedures

Cited by 18 publications

References 29 publications

Systematic review of the use of “magnitude-based inference” in sports science and medicine

Systematic review of the use of “magnitude-based inference” in sports science and medicine

Systematic Review of the use of “Magnitude-Based Inference” in Sports Science and Medicine

Monitoring training and recovery responses with heart rate measures during standardized warm-up in elite badminton players

Contact Info

Product

Resources

About