Evaluating Methods to Estimate the Effect of State Laws on Firearm Deaths: A Simulation Study

Abstract

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited. Permission is given to duplicate this document for personal use only, as long as it is unaltered and complete. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial use. For information on reprint and linking permissions, please visit www.rand.org/pubs/permissions. The RAND Corporation is a research organization that develops solutions to public policy challenges to help make communities throughout the world safer and more secure, healthier and more prosperous. RAND is nonprofit, nonpartisan, and committed to the public interest. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.

Variation in Performance of Commonly Used Statistical Methods for Estimating Effectiveness of State-Level Opioid Policies on Opioid-Related Mortality

“…Findings from our simulations on opioid-related mortality are highly similar to findings for total firearm deaths considered in Schell, Griffin, and Morral (2019). The models considered here are virtually identical to the models considered in the Schell, Griffin, and Morral (2019) study, though we expand the simulation scenarios to consider performance across a wider range of assumed effect sizes of the law (5% to 25% versus 3% in Schell, Griffin, and Morral (2019)). In their case, one model proved optimal across all performance metrics considered: the negative binomial AR model, whereas we found in favor of two set of models of which the negative binomial AR model is one.…”

“…The simulation design builds directly from prior work that compared statistical methods for evaluating the impact of state laws on firearms deaths (Schell, Griffin, and Morral 2019). For each simulation iteration, we selected a random subset of k states to be the policy/treated group (i.e., it = 1 at some point in the study period), with remaining states serving as the comparison group (i.e., it = 0 for the entire study period).…”

“…There are a growing number of studies highlighting challenges and limitations to using DID models, particularly when the key assumptions of the DID model do not hold (Blundell and Costa Dias 2009, Daw and Hatfield 2018a, b, Ryan, Burgess, and Dimick 2015or when a study involves small sample sizes (Brewer, Crossley, and Joyce 2017). Additionally, it has been well-established that standard error corrections that attempt to adjust for violations of the assumed independence of the repeated measures in longitudinal datasets are needed to obtain accurate Type I error rates (Abhay, Donohue III, and Zhang 2014, Bertrand, Duflo, and Mullainathan 2004, Donald and Lang 2007, Helland and Tabarrok 2004, Schell, Griffin, and Morral 2019. Despite the wealth of knowledge concerning best practices and the challenges of using DID models in various settings, there have been no comphrenensive examinations of their performance across settings realistic for state opioid policy evaluation.…”

Prescription drug monitoring programs: Assessing the association between “best practices” and opioid use in Medicare

“…Although models commonly used in policy analysis often include both state and time fixed effects, emerging research on the effect of state laws on firearms deaths indicates the best model may depend on how the exposure is operationalized and whether autoregressive terms are used . Among nonautoregressive models, Schell et al found that using time fixed effects with adjustment to the standard error via GEE was competitive with the best performing model which uses time fixed effects, change levels coding of the law exposure variable, lags of the outcome, and no adjustment to the standard error.…”

Indirect simultaneity

“…Another improvement is state-specific trend variables. Perhaps the most important controls are lagged dependent variables (Schell, Griffin, & Morral, 2018;Spelman, 2016). They control for missing variables that affect the dependent variable through its lag.…”