Balancing Treatment and Control Groups in Quasi‐Experiments: An Introduction to Propensity Scoring

Abstract

Organizational and applied sciences have long struggled with improving causal inference in quasi-experiments. We introduce organizational researchers to propensity scoring, a statistical technique that has become popular in other applied sciences as a means for improving internal validity. Propensity scoring statistically models how individuals in a quasi-experiment have been assigned to conditions in order to estimate treatment effects among individuals with approximately equal probabilities of receiving the treatment. If propensity scores are created from relevant covariates, matching on the propensity score makes treatment assignment ignorable and approximates a true experimental design. We illustrate how matching on the propensity score can be applied by using 2 examples: examining the effects of online instruction and estimating the benefits of preparatory coaching for the SAT. In both cases, propensityscoring methods effectively reduced inequivalence between treatment and control groups on many variables. Propensity scoring stands out as a valuable technique capable of improving causal inference from many of organizational research's quasi-experiments.Organizational researchers are frequently called on to evaluate the effectiveness of organizational interventions, and circumstances frequently impose real-world limitations on the designs available to make these evaluations. Many such designs fall within the broad category of "quasiexperiments" because, although some individuals receive the intervention (e.g., a new compensation system or a new training course) while others We thank

Effects of Organizationally Endorsed Coaching on Performance and Validity of Situational Judgment Tests

“…Candidates motivated to seek coaching will do so. Thus, the use of control variables, as in the present study, or propensity scores, as in the case of Lievens et al (2012) and Connelly, Sackett, and Waters (2013), appear to be the best available strategies for addressing the selection bias issue.…”

Analytical mindsets in turnover research

“…Another possible avenue is to identify naturally occurring field experiments, such as McNatt and Judge's () study of the effects of a self‐efficacy intervention in reducing turnover. Turnover scholars may also adopt the practice of propensity scoring in quasi‐experiments as a way to balance treatment and control groups when randomization is not feasible (Connelly, Sackett, & Waters, ). It might also be useful to reconsider the value of lab research.…”

Learning in the Twenty-First-Century Workplace

“…Many researchers have been unable to evaluate change in their assessments of the effectiveness of learning interventions because of an inability to randomly assign study participants to conditions. Connelly et al (2013) show how propensity scoring can be used in quasiexperimental designs to improve causal inference.…”