“Life Doesn't Happen at the Between-Person Level,” or a Cautionary Note on Generating Scientific Inferences Through Meta-Analyses

Abstract

The implicit philosophy for how research and practice in industrial and organizational (I-O) psychology has pursued inferences about our field's core phenomena has largely been based on a nomothetic, variable-based, and aggregate/“large-sample” ideal. As Tett, Hundley, and Christiansen (2017) expertly highlight, there are more insightful means for drawing inferences about the nature of such aggregate relationships based on meta-analytic techniques than the current practice in the organizational sciences. However, the motivating force behind our commentary has less to do with the issues raised by Tett et al. (2017) concerning the practice of using meta-analysis for purposes of validity generalization and more to do with the practice of using meta-analysis for purposes of scientific inference. Between-person philosophies in which the end-goal is to identify general conclusions that apply to the aggregate (cf. Hanges & Wang, 2012) have historically guided our scientific inferences and have supported the proliferation of meta-analytic techniques (including what Tett et al. describe as tertiary analyses based on such findings). These philosophies have led to a dearth of understanding at the within-person and social system levels—the levels at which most of our meaningful phenomena exist (e.g., Hamaker, 2012; Von Bertalanffy, 1950). Learning, performing, decision making, communicating, sense-making, feeling/expressing emotion: These are the concepts that drive the lived experiences of individuals both inside and outside of the workplace, and all are vulnerable to being misunderstood or misinterpreted by focusing only on aggregate evidence at the between-person level. Consequently, we wish to first supplement Tett et al.’s recommendations for drawing generalizability inferences in meta-analysis and suggest a “pre-emptive” question (i.e., Question 0) to the list of four they advance in their focal article.