ML-DEs: A program for designing efficient multilevel studies

Abstract

The multilevel model is increasingly used as a flexible tool in the statistical analysis of dependent behavioral research data. A drawback of this model's flexibility is that it complicates designing the study. For example, an important additional consideration in the design of a multilevel study is choosing the number and the size of the clusters to sample to ensure sufficient efficiency as quantified by precision, bias, or statistical power. To help researchers in designing their multilevel study, a user-friendly simulation tool is introduced (Multilevel Design Efficiency Using Simulation, or ML-DEs), also allowing for design questions that have not been dealt with analytically in the literature, while avoiding complex specifications of simulation studies. ML-DEs generates MLwiN macros for running the simulations and handles its output using R scripts to compare the designs' efficiencies for both fixed and random parameters, allowing for small sample sizes, unbalanced data, and more than two levels.Various designs of varying efficiency can be used to answer specific research questions. They may differ in many ways: in the amount of information they provide as quantified by, for example, the accuracy of estimation (Kelley & Maxwell, 2003) and/or the power for statistical testing (J. Cohen, 1988); or alternatively, in how many resources are required to obtain a certain amount of information. Carefully designing studies to be efficient is especially important for behavioral research, which often has to deal with small effects and considerable uncertainty about the estimates (Pillemer, 1984).Whereas it is well understood how to increase the design efficiency for simple models (Howell, 2005), for more complex models and their corresponding analyses this relation between design and efficiency becomes much less straightforward, as with, for example, multilevel data analyses (Snijders, 2005). The multilevel statistical model is increasingly used, because it elegantly takes into account dependencies among observations that exist because of multistage sampling, with sampled observations embedded within clusters; or because repeated measurements were performed, with a sequence of observations embedded within units. Similarly, meta-analyses and multivariate analyses can also be dealt with using multilevel analyses (Raudenbush, 1988;Van den Noortgate & Onghena, 2003.Multilevel design efficiency has already been studied analytically, as well as in simulation studies. Whereas the analytical studies only deal with quite simple models and imply several assumptions, the results of the simulation studies may be difficult to generalize about, because data are generated from a specific model and specific parameter values. In response to both problems, this article will present a user-friendly tool named Multilevel Design Efficiency Using Simulation (ML-DEs), which allows applied researchers to derive more efficient multilevel designs for their own research. Unlike existing programs that use analytical results, this tool will allow...

The Three-Level Synthesis of Standardized Single-Subject Experimental Data: A Monte Carlo Simulation Study

“…If a researcher is interested in the power when including other values for the different conditions, several tools are available. For instance, Cools, Van den Noortgate, and Onghena (2008) developed a user-friendly tool, MultiLevel Design Efficiency Using Simulation (ML-DEs), that can be used for calculating the number of units needed at each level to obtain a power larger than .80. The user has to specify the model of interest (e.g., the number of levels, the number of variables at each level, the covariance structures, and the parameter values) and the number of sample sizes at each level for which data will be generated and will receive power and accuracy estimates for the parameters of interest.…”

Selection of the Number of Participants in Intensive Longitudinal Studies: A User-friendly Shiny App and Tutorial to Perform Power Analysis in Multilevel Regression Models that Account for Temporal Dependencies

“…Since the data have been randomly generated, the parameter estimates and the test results will vary across the data sets. Hence, we can compute the power as the proportion of simulated data sets for which the null hypothesis about the parameter(s) of interest has been rejected (see, e.g., Arend & Schäfer 2019, Astivia et al 2019, Bolger 2011, Browne et al 2009, Cools et al 2008, Green & MacLeod 2016, Landau & Stahl 2013, Lane & Hennes 2018, Maas & Hox 2005, Mathieu et al 2012, Zhang & Wang 2009, Zhang 2014. Performing these calculations while varying the number of participants allows us to determine the number of participants necessary to reach a predetermined amount of power (e.g., 80%).…”

“…This requires that one includes either serially correlated errors or the lagged outcome variable as a predictor in the multilevel models. Although there are several resources available to perform power analyses for multilevel models (e.g., Arend & Schäfer 2019, Browne et al 2009, Cools et al 2008, Green & MacLeod 2016, Hedeker et al 1999, Landau & Stahl 2013, Lane & Hennes 2018, Mathieu et al 2012, Raudenbush 1997, Raudenbush & Liu 2001, Snijders & Bosker 1993, Zhang & Wang 2009, Zhang 2014), these do not account for the temporal dependencies that characterize IL data.…”

Depressive symptoms in adolescence: The role of perceived parental support, psychological control, and proactive control in interaction with 5-HTTLPR

“…Ancestry was assessed using the grandparents' country of birth (88.8% European, 6.5% non-European Mediterranean, 2.2% non-European, non-Mediterranean ancestry, and 2.5% missing). G*Power 3 and ML-DEs [57,58], two statistical programs to conduct power analysis, had shown that a sample size of 800 has more than 80% power for detecting small environmental effects, genetic effects, and G Â E interactions, each accounting for only 1% of the variance with a = .05. Because we had no prior information available on the 5-factor model for parenting [20], we could only estimate power using the proportion of explained variance.…”