The Minimum Detectable Difference (MDD) Concept for Establishing Trust in Nonsignificant Results: A Critical Review

Abstract

Current regulatory guidelines for pesticide risk assessment recommend that nonsignificant results should be complemented by the minimum detectable difference (MDD), a statistical indicator that is used to decide whether the experiment could have detected biologically relevant effects. We review the statistical theory of the MDD and perform simulations to understand its properties and error rates. Most importantly, we compare the skill of the MDD in distinguishing between true and false negatives (i.e., type II errors) with 2 alternatives: the minimum detectable effect (MDE), an indicator based on a post hoc power analysis common in medical studies; and confidence intervals (CIs). Our results demonstrate that MDD and MDE only differ in that the power of the MDD depends on the sample size. Moreover, although both MDD and MDE have some skill in distinguishing between false negatives and true absence of an effect, they do not perform as well as using CI upper bounds to establish trust in a nonsignificant result. The reason is that, unlike the CI, neither MDD nor MDE consider the estimated effect size in their calculation. We also show that MDD and MDE are no better than CIs in identifying larger effects among the false negatives. We conclude that, although MDDs are useful, CIs are preferable for deciding whether to treat a nonsignificant test result as a true negative, or for determining an upper bound for an unknown true effect.

Exposure to a youth-led sexual violence prevention program among adolescents: The impact of engagement.

“…We ensured a large sample by choosing an appropriately large school district; however, our sample size was greatly impacted by uncontrollable factors such as how many parents gave permission, how many students moved away from the district, and absenteeism on survey days. After conducting analyses, we conducted a minimum detectable effect analysis (Juras et al, 2016; Mair et al, 2020). These analyses indicated that the study was appropriately powered for the three main outcomes presented in the current analyses (overall peer violence perpetration, overall victimization, and sexual victimization).…”

Does learning history shape the associability of outcomes? Further tests of the outcome predictability effect

“…Therefore, and in addition to justified questions about the appropriate estimate of the effect size as well as assumptions about the correlations between repeated measures, the results of our initial analyses of sample sizes and power have to be interpreted cautiously. Calculating confidence intervals around the estimated effect, η 2 p , can help determine whether a nonsignificant result indicates the true absence of an effect rather than a lack of power [56], as the true value of the population effect lies within this interval [57]. The lower bound for most of the experiments includes values of zero and therefore no effect at all (Table 4, [58,59]), which is in line with null hypothesis significance testing of the ANOVAs.…”

Evaluation of levocetirizine in beagle dog and cynomolgus monkey telemetry assays: Defining the no QTc effect profile by timepoint and concentration‐QTc analysis

“…A statistical analysis was conducted to determine the minimal detectable difference (MDD) in the QTc interval and calculated as the half‐width of the 95% CI. The MDD value represents the smallest difference between group means that would be detectable in the t ‐test at a probability level <0.05 (e.g., the border between significance and nonsignificance) 23 . The inclusion of an in‐study statistical metric is a best practice consideration for new cardiovascular telemetry studies to quantify QTc assay sensitivity and for potential comparison to prior studies 18,24 …”