Sample Size Justification

Abstract

An important step when designing a study is to justify the sample size that will be collected. The key aim of a sample size justification is to explain how the collected data is expected to provide valuable information given the inferential goals of the researcher. In this overview article six approaches are discussed to justify the sample size in a quantitative empirical study: 1) collecting data from (an)almost) the entire population, 2) choosing a sample size based on resource constraints, 3) performing an a-priori power analysis, 4) planning for a desired accuracy, 5) using heuristics, or 6) explicitly acknowledging the absence of a justification. An important question to consider when justifying sample sizes is which effect sizes are deemed interesting, and the extent to which the data that is collected informs inferences about these effect sizes. Depending on the sample size justification chosen, researchers could consider 1) what the smallest effect size of interest is, 2) which minimal effect size will be statistically significant, 3) which effect sizes they expect (and what they base these expectations on), 4) which effect sizes would be rejected based on a confidence interval around the effect size, 5) which ranges of effects a study has sufficient power to detect based on a sensitivity power analysis, and 6) which effect sizes are plausible in a specific research area. Researchers can use the guidelines presented in this article to improve their sample size justification, and hopefully, align the informational value of a study with their inferential goals.

Music Student’s Approach to the Forced Use of Remote Performance Assessments

“…Our findings provide further insight into the influence of COVID-19 on student experience. However, given the scarcity of research including musicians during this period (Philippe et al, 2020;Spiro et al, 2020;Tymoszuk et al, 2021), the sample size being determined by accessibility and time constraints (as is common in psychology research; Lakens, 2021) and the influence posed by this pandemic, we encourage caution when interpreting the study's p-values in a generalizable way. As such, future researchers must extend the current findings (e.g., comparing the impact of teaching and student experience before, during and after COVID-19).…”

Group Sequential Designs: A Tutorial

“…We hope this tutorial increases the uptake of group sequential designs in psychological science, thereby increasing the efficiency of data collection. As an increasing number of journals are starting to expect a sample size justification for each study that is published, sequential designs offer a useful extension to current research practices, especially when there is large uncertainty about the effect size one expects (Lakens, 2021). Instead of an a-priori power analysis for an expected effect size, researchers can determine the smallest effect size they would be interested in detecting, determine the maximum sample size they will collect, and perform interim analyses that allow them to stop early when the null hypothesis can be rejected, or the smallest effect size of interest can be rejected.…”

Impact of proprioception on the perceived size and distance of external objects in a virtual action task

“…As estimation of effect sizes from small sample sizes could be inaccurate, we also computed confidence intervals (80%) for the expected effects based on the pilot data set (see also, e.g., Cocks & Torgerson, 2013;Lakens, 2021). These values amounted to 0.56 ± .76 mm for the size effect and 6.46 ± 4.14 mm for the distance effect.…”