Finding the right power balance: Better study design and collaboration can reduce dependence on statistical power

Nakagawa, Shinichi; Lagisz, Malgorzata; Yang, Yefeng; Drobniak, Szymon M.

doi:10.1371/journal.pbio.3002423

“…We urge that funding bodies and ethics committees recognise that if they ask for sex inclusion, they will need to allow researchers to have more subjects and budget to gain robust statistical inference and, therefore, robust conclusions (cf. [ 14 ]). There is no way around this biological and statistical reality.…”

Section: Discussionmentioning

confidence: 99%

Realism and robustness require increased sample size when studying both sexes

Drobniak,

Lagisz,

Yang

et al. 2024

PLoS Biol

Self Cite

1

0

View full text Add to dashboard Cite

Female subjects have been historically excluded from biomedicine and other related areas of study [1]. Such exclusion has disadvantaged females and prevented a fuller understanding of biology. Therefore, in 2016, the National Institute of Health (NIH) mandated that all NIH-funded animal and human studies consider sex as a biological variable (e.g., [2]). Yet, sex as a biological variable has not been welcomed with open arms, most likely because many researchers believe they need to increase the overall sample size with 2 sexes compared to using only 1 sex (e.g., [2]). Recently, Philips and colleagues published a PLoS Biology article titled "Statistical simulations show that scientists need not increase overall sample size by default when including both sexes in in vivo studies" [3]. As indicated in their title, the authors have concluded and recommended no increase in sample size with both sexes, which was based on a set of simulations exploring a simple but-as they claim-likely biological scenario. Their conclusion is great news for researchers who feared coping with increased experiment sizes and costs.However, Philips and colleagues have assumed homoscedasticity between the 2 sexes, meaning variances or standard deviations of the sexes are the same throughout their simulations. Here, we first explain why such an assumption is biologically unrealistic and why heteroscedasticity between 2 sexes should be the norm rather than the exception by pointing out a wealth of empirical evidence and evolutionary arguments. We then show the results from a simulation study expanding Philips and colleagues' work by incorporating heteroscedasticity. Our results clearly indicate that we need to increase the overall sample size to have robust statistical inference. Further, we provide statistical recommendations to deal with heteroscedasticity. We also briefly touch on what funding agencies and ethics committees can do, given our results.

show abstract

“…The latter factor matches the serologic status of CCP recipients: patients with a reduced or absent antibody response against SARS-CoV-2, such as immunocompromised patients, respond better to high-titer CCP therapy [55]. In addition to the timing and dosing of CCP, there are other key determining factors of CCP effectiveness, among which the study design must be mentioned, in particular the sample size estimation [62,63]. Sample size calculation is an essential component of a study protocol.…”

Section: Discussionmentioning

confidence: 99%

Exploring Study Design Foibles in Randomized Controlled Trials on Convalescent Plasma in Hospitalized COVID-19 Patients

Franchini,

Mengoli,

Casadevall

et al. 2024

Life

0

View full text Add to dashboard Cite

Background: Sample size estimation is an essential step in the design of randomized controlled trials (RCTs) evaluating a treatment effect. Sample size is a critical variable in determining statistical significance and, thus, it significantly influences RCTs’ success or failure. During the COVID-19 pandemic, many RCTs tested the efficacy of COVID-19 convalescent plasma (CCP) in hospitalized patients but reported different efficacies, which could be attributed to, in addition to timing and dose, inadequate sample size estimates. Methods: To assess the sample size estimation in RCTs evaluating the effect of treatment with CCP in hospitalized COVID-19 patients, we searched the medical literature between January 2020 and March 2024 through PubMed and other electronic databases, extracting information on expected size effect, statistical power, significance level, and measured efficacy. Results: A total of 32 RCTs were identified. While power and significance level were highly consistent, heterogeneity in the expected size effect was relevant. Approximately one third of the RCTs did not reach the planned sample size for various reasons, with the most important one being slow patient recruitment during the pandemic’s peaks. RCTs with a primary outcome in favor of CCP treatment had a significant lower median absolute difference in the expected size effect than unfavorable RCTs (20.0% versus 33.9%, P = 0.04). Conclusions: The analyses of sample sizes in RCTs of CCP treatment in hospitalized COVID-19 patients reveal that many underestimated the number of participants needed because of excessively high expectations on efficacy, and thus, these studies had low statistical power. This, in combination with a lower-than-planned recruitment of cases and controls, could have further negatively influenced the primary outcomes of the RCTs.

show abstract

“…We urge that funding bodies and ethics committees recognize that if they ask for sex inclusion, they will need to allow researchers to have more subjects and budget to gain robust statistical inference and, therefore, robust conclusions (cf. [14]). There is no way around this biological and statistical reality.…”

Section: Discussionmentioning

confidence: 99%

Realism and robustness require increased sample size when studying both sexes

Drobniak,

Lagisz,

Yang

et al. 2023

Preprint

0

View full text Add to dashboard Cite

Female subjects have been historically excluded from biomedicine and other related areas of study [1]. Such exclusion has disadvantaged females and prevented a fuller understanding of biology. Therefore, in 2016, the National Institute of Health (NIH) mandated that all NIH-funded animal and human studies consider sex as a biological variable (e.g., [2]). Yet, sex as a biological variable has not been welcomed with open arms, most likely because many researchers believe they need to increase the overall sample size with two sexes compared to using only one sex (e.g., [2]). Recently, Philips and colleagues published a PLoS Biology article titled “Statistical simulations show that scientists need not increase overall sample size by default when including both sexes in in vivo studies” [3]. As indicated in their title, the authors have concluded and recommended no increase in sample size with both sexes, which was based on a set of simulations exploring a simple but – as they claim – likely biological scenario. Their conclusion is great news for researchers who feared coping with increased experiment sizes and costs.

show abstract

Finding the right power balance: Better study design and collaboration can reduce dependence on statistical power

Cited by 6 publications

References 107 publications

Realism and robustness require increased sample size when studying both sexes

Realism and robustness require increased sample size when studying both sexes

Exploring Study Design Foibles in Randomized Controlled Trials on Convalescent Plasma in Hospitalized COVID-19 Patients

Realism and robustness require increased sample size when studying both sexes

Contact Info

Product

Resources

About