Meta-Analytic Findings of the Self-Controlled Motor Learning Literature: Underpowered, Biased, and Lacking Evidential Value

Abstract

The self-controlled motor learning literature consists of experiments that compare a group of learners who are provided with a choice over an aspect of their practice environment to a group who are yoked to those choices. A qualitative review of the literature suggests an unambiguous benefit from self-controlled practice. A meta-analysis was conducted on the effects of self-controlled practice on retention test performance measures with a focus on assessing and potentially correcting for selection bias in the literature, such as publication bias and p-hacking. First, a naïve random effects model was fit to the data and a moderate benefit of self-controlled practice, g = .44 (k = 52, N = 2061, 95% CI [.31, .56]), was found. Second, publication status was added to the model as a potential moderator, revealing a significant difference between published and unpublished findings, with only the former reporting a benefit of self-controlled practice. Third, to investigate and adjust for the impact of selectively reporting statistically significant results, a weight-function model was fit to the data with a one-tailed p-value cutpoint of .025. The weight-function model revealed substantial selection bias and estimated the true average effect of self- controlled practice as g = .107 (95% CI [.047, .18]). P-curve analyses were conducted on the statistically significant results published in the literature and the outcome suggested a lack of evidential value. Fourth, a suite of sensitivity analyses were conducted to evaluate the robustness of these results, all of which converged on trivially small effect estimates. Overall, our results suggest the benefit of self-controlled practice on motor learning is small and not currently distinguishable from zero.

On the Reproducibility of Power Analyses in Motor Behavior Research

“…When power analyses were reported, they were rarely reproducible without making assumptions, and even then, most power analyses could not be reproduced. Meanwhile, there is growing evidence that the average power among motor behavior studies is low, making the literature vulnerable to more severe bias from various selective reporting mechanisms (e.g., Lohse et al, 2016;McKay, Yantha, et al, 2022;Mesquida et al, 2022). Here, we argue that power analyses can easily be reported in a reproducible fashion and doing so is a progressive step toward improved research quality overall.…”

“…A lack of thoroughly reported and vetted power analyses contributes to the proliferation of underpowered studies, which combined with selection for significant results threatens the credibility of our literature. The impact of low power and selection bias is well illustrated by the growing body of metascience calling into question the reliability of research paradigms long considered robust (Carter et al, 2015;e.g., Maier et al, 2022;Vohs et al, 2021), such as self-controlled practice in the motor learning domain (McKay, Yantha, et al, 2022). In a recent metaanalysis, McKay and colleagues estimated the benefit to motor learning of giving learners control over an aspect of their environment is trivially small, if existent, after correcting for publication bias.…”

“…The analysis found that despite an average benefit of g = .54 in the published literature, the true effect of enhanced expectancies is likely much smaller, if it exists at all. The studies examined in these meta-analyses had median sample sizes of n = 14 (McKay, Yantha, et al, 2022) and n = 18 (McKay, Bacelar, et al, 2022), requiring effects larger than g = .8 to achieve significance with an independent t test. Therefore, selectively publishing significant results in these literatures meant publishing an abundance of large effects, making it possible for even null effects to appear moderately beneficial on average.…”

“…For example, the average power of experiments (k = 75) that compared a reduced frequency of feedback with a 100% frequency was estimated to be 27%, again with substantial uncertainty . Even lower average power estimates of 6% were reported in meta-analyses of enhanced expectancies (Bacelar et al, 2022;McKay, Bacelar, et al, 2022) and self-controlled practice (McKay, Yantha, et al, 2022), with an upper bound estimate of 13%. Despite having a low probability of observing a significant result a priori, positive results in these literatures have been much more frequent than expected.…”

Low Prevalence of A Priori Power Analyses in Motor Behavior Research

“…We focused on motor behavior research as recent meta-analyses have reported evidence of both underpowered research and substantial reporting bias in motor learning and sports science (Lohse et al, 2016;2022a, b;Mesquida et al, 2022). For example, a meta-analysis of the self-controlled motor learning literature estimated the average power of all studies conducted was 6%, while 48% of studies reported significant results on the focal measure (McKay et al, 2022b). Other studies have estimated average power ranging from 20% (McKay et al, 2022a) to 50% (Mesquida et al, 2022), with significant indications of reporting bias.…”