On Ignoring the Random Effects Assumption in Multilevel Models: Review, Critique, and Recommendations

Abstract

Entities such as individuals, teams, or organizations can vary systematically from one another. Researchers typically model such data using multilevel models, assuming that the random effects are uncorrelated with the regressors. Violating this testable assumption, which is often ignored, creates an endogeneity problem thus preventing causal interpretations. Focusing on two-level models, we explain how researchers can avoid this problem by including cluster means of the Level 1 explanatory variables as controls; we explain this point conceptually and with a large-scale simulation. We further show why the common practice of centering the predictor variables is mostly unnecessary. Moreover, to examine the state of the science, we reviewed 204 randomly drawn articles from macro and micro organizational science and applied psychology journals, finding that only 106 articles—with a slightly higher proportion from macro-oriented fields—properly deal with the random effects assumption. Alarmingly, most models also failed on the usual exogeneity requirement of the regressors, leaving only 25 mostly macro-level articles that potentially reported trustworthy multilevel estimates. We offer a set of practical recommendations for researchers to model multilevel data appropriately.

Endogeneity: A Review and Agenda for the Methodology-Practice Divide Affecting Micro and Macro Research

“…Our review aims to enhance understanding of endogeneity - an issue that poses serious implications for interpretation of study outcomes - by organizing the vast literature on the sources of bias and methodological solutions (e.g., Antonakis et al, 2019; Bhave, 2014; Maynard, Luciano, D’Innocenzo, Mathieu, & Dean, 2014). As we outlined above, definitional and terminology differences across this literature hinder direct comparisons on the term endogeneity alone.…”

“…A few caveats of fixed effects are notable. First, fixed effects do not fix all endogeneity concerns, but they do work in situations where the omitted variable is constant for all observations with the same fixed effect (Antonakis, Bastardoz, & Rönkkö, 2019). Second, fixed effect analyses assess within effects not between effects (for a discussion, see Certo, Withers, & Semadeni, 2017).…”

Working 9‐to‐5? A review of research on nonstandard work schedules

“…In recent years, though, researchers have increasingly adopted a within‐person approach to understand a variety of organizational phenomena, such as emotions, goals, motivation, coping behavior, creativity, citizenship behavior, work–family conflict, and stressors, among others (Fisher & To, 2012). This line of inquiry generally relies on experience sampling methodology, and it is especially useful for understanding the dynamics of within‐person processes like mood, affect, behavior, work events, and other phenomena that vary on a daily or weekly basis (for an example from this review, see Spieler et al, 2017); further, these methodologies can explore relations relative to participants' baseline which allows researchers to control for individual differences (Antonakis, Bastardoz, & Rönkkö, 2019). They likewise allow researchers to explore the cross‐level effects of stable personal or contextual factors on within‐person processes.…”

A diary study on shared leadership, team work engagement, and goal attainment

“…In order to remove between-team differences that may influence the results, we centered our data on the respective team mean and added the team mean on Level 3. This procedure allows to estimate the within-team relationships without confounding effects of between-team differences (see Antonakis et al, 2019). Nevertheless, day-level formal leadership may still be a plausible alternative to explain our pattern of results, although recent meta-analyses suggested that shared leadership may be important for team effectiveness over the A DIARY STUDY ON SHARED LEADERSHIP 27 influence of formal leadership (Nicolaides et al, 2014;D.…”

“…To advance the nomological network of shared leadership and to get a more nuanced view on its within-team relationships (Kelemen et al, 2020), it hence may be useful to examine in how far the daily fluctuations in shared leadership are related to day-level progress in attaining collective goals. Additionally, by simultaneously considering the within-and between-team consequences of shared leadership from a multilevel perspective, this study accounts for possible endogeneity bias for the withinteam relationships due to between-team differences (Antonakis, Bastardoz, & Rönkkö, 2019).…”