Know your population and know your model: Using model-based regression and poststratification to generalize findings beyond the observed sample.

Abstract

Psychology research often focuses on interactions, and this has deep implications for inference from nonrepresentative samples. For the goal of estimating average treatment effects, we propose to fit a model allowing treatment to interact with background variables and then average over the distribution of these variables in the population. This can be seen as an extension of multilevel regression and poststratification (MRP), a method used in political science and other areas of survey research, where researchers wish to generalize from a sparse and possibly nonrepresentative sample to the general population. In this article, we discuss areas where this method can be used in the psychological sciences. We use our method to estimate the norming distribution for the Big Five Personality Scale using open source data. We argue that large open data sources like this and other collaborative data sources can potentially be combined with MRP to help resolve current challenges of generalizability and replication in psychology.

Validity and usefulness of the student-athletes’ motivation toward sport and academics questionnaire: a Bayesian multilevel approach

“…Given the debate concerning the limitations and inappropriateness of null-hypothesis testing, ''statistical significance'' and using p-values (Amrhein & Greenland, 2018;McShane et al, 2019), the Bayesian statistics was deemed appropriate to provide a natural approach to account for different sources of inferential uncertainty (Kennedy & Gelman, 2020). Furthermore, in the present study, the analytical approach to validate a questionnaire using a multilevel modeling framework was considered appropriate to deal with common limitations of sports psychology research, such as noisy measurements, between-individuals heterogeneity, complex interactions between outcomes, and non-representative and imbalanced samples.…”

“…Given attrition expected in longitudinal observations, a Bayesian multilevel regression modeling and poststratification were used to predict the changes applied to all observations in the cross-sectional data. Bayesian multilevel regression and poststratification allow for improved estimations of small and sparse group data (in the present study, the longitudinal observations) and consequently predicts a target population (in the present study, the cross-sectional observations) (Gelman & Little, 1997;Park, Gelman & Bafumi, 2004;Kennedy & Gelman, 2020).…”

“…Indeed, psychology research deals with complex interactions, noisy measurements, often expected between-individuals heterogeneity, and non-representative and imbalanced samples. To account for different sources of inferential uncertainty, Bayesian methods allow combining the information known before seeing the data (i.e., the prior uncertainty concerning a parameter or hypothesis expressed as a probability distribution) with what is learned from the observed data (i.e., the likelihood of the data conditioned on the parameter or hypothesis) to update knowledge expressed as the posterior distribution (Kennedy & Gelman, 2020;Lee & Wagenmakers, 2013).…”

Revisiting the Relative Age Effect From a Multidisciplinary Perspective in Youth Basketball: A Bayesian Analysis

“…As an alternative to excluding players, we modeled our data using Bayesian multilevel regression and poststratification (Gelman and Little, 1997). The technique allows us to estimate the outcomes in small groups using varying effects for individual-level predictors such as gender, age group, maturity status, or onset of deliberate practice, that take on multiple levels in the data (Kennedy and Gelman, 2020) In the second part of the method, we use the multilevel model estimates to predict the players' outcomes for groups defined in a post-stratification dataset (i.e., gender, birth quarter, age group, maturity status, and the onset of deliberate basketball practice). The post-stratification table has an observation corresponding to each group defined for all combinations of the variables included in the model.…”

“…Hence, the method allows us to take full use of all available data. Lastly, we prefer Bayesian methods as it provides a natural approach to account for different sources of inferential uncertainty (Kennedy and Gelman, 2020).…”