Deviating from the Mean: The Declining Significance of Significance

Abstract

Most of the methods we use in criminology to infer relationships are based on mean values of distributions. This essay explores the historical origins of this issue and some counterproductive consequences: relying too heavily on sampling as a means of ensuring “statistical significance”; ignoring the implicit assumptions of regression modeling; and assuming that all data sets reflect a single mode of behavior for the entire population under study. The essay concludes by suggesting that we no longer “make do” with the standard methodologies used to study criminology and criminal justice, and recommends developing categories that more accurately reflect behavior and groupings than the ones we currently use; looking at alternative sources of data, including qualitative data such as narrative accounts; and developing alternative methods to extract and analyze the data from such sources.

Deterrence, Firearm Arrests, and Subsequent Shootings: A Micro-Level Spatio-Temporal Analysis

“…Given the newness of this technique, it is not certain whether Monte Carlo procedures for generating expected distributions are likely to generate an inflated experiment-wise alpha rate across an entire set of cells. In order to counterbalance this potential concern, attention was focused just on the nine cells in the three time buffers and distances soonest after and closest to an initiating event, which also were the most theoretically relevant given earlier work; ratios which were not well patterned were not emphasized; and considerable attention was devoted to the patterning of the ratios within these nine cells, following Maltz's (1994) advice.…”

Looking Back to Move Forward

“…JRCD has published numerous influential articles related to measurement and theoretical development-sometimes looking at the two in tandem (e.g., Farnworth et al 1994). From Gold's (1966) and Hardt and Peterson-Hardt's (1977) articles on measurement of self-reported delinquency within the journal's first 15 years, which still resonate in more recent work assessing the validity of self-reports (e.g., Kim, Fenrich, and Wislar 2000;Paschall, Ornstein, and Flewelling 2001;Wells and Rankin 1995), to Bernard's (1990) overview of theoretical progress (or lack thereof) in criminology, works on measuring self-control within the general theory of crime (see Grasmick et al 1993;Hirschi and Gottfredson 1993;Keane, Maxim, and Teevan 1993), Maltz's (1994) questions about how our methods do and do not serve our substantive goals, Morris and Slocum's (2010) work studying the life events calendar and Boman and colleagues' (2012) consideration of how peer delinquency is measured, this journal has clearly paid attention to important issues in measurement (see also Armstrong, Lee, and Armstrong 2009;Evans and Scott 1984). In the process, authors have both implicitly and explicitly weighed the implications of key measurement issues for knowledge development in the context of individual studies and areas of research more broadly.…”

“…These analytic advancements, which generally seek to improve the identification of estimates, are extremely important and warranted, but they ultimately provide a better test of the relationship among variables, which do not, ipso facto, generalize well to the latent concepts supposedly underlying them. In other words, most of these emerging analytic tools do not inherently improve construct validity, which is more closely aligned with the design and data collection stages of scientific process, as they have a tendency to shift the main objective of research from measurement to statistical analysis (Maltz 1994). This is unfortunate as, regardless of the statistical tools at our disposal, design and data collection are the phases in the research process most important in matters related to causal inference (Rubin 2008).…”

An information theory approach to hypothesis testing in criminological research

“…First is the reification of statistical significance as the most important outcome of quantitative research (Maltz 1994). Replicating analysis of papers published in the American Economic Review (McCloskey and Ziliak 1996;Ziliak and McCloskey 2004), Bushway and colleagues show that criminologists similarly more prominently report statistical significance than effect size.…”