The influence of base rates on correlations: An evaluation of proposed alternative effect sizes with real-world data

Abstract

Correlations are the simplest and most commonly understood effect size statistic in psychology. The purpose of the current paper was to use a large sample of real-world data (109 correlations with 60,415 participants) to illustrate the base rate dependence of correlations when applied to dichotomous or ordinal data. Specifically, we examined the influence of the base rate on different effect size metrics. Correlations decreased when the dichotomous variable did not have a 50 % base rate. The higher the deviation from a 50 % base rate, the smaller the observed Pearson's point-biserial and Kendall's tau correlation coefficients. In contrast, the relationship between base rate deviations and the more commonly proposed alternatives (i.e., polychoric correlation coefficients, AUCs, Pearson/Thorndike adjusted correlations, and Cohen's d) were less remarkable, with AUCs being most robust to attenuation due to base rates. In other words, the base rate makes a marked difference in the magnitude of the correlation. As such, when using dichotomous data, the correlation may be more sensitive to base rates than is optimal for the researcher's goals. Given the magnitude of the association between the base rate and point-biserial correlations (r = −.81) and Kendall's tau (r = −.80), we recommend that AUCs, Pearson/Thorndike adjusted correlations, Cohen's d, or polychoric correlations should be considered as alternate effect size statistics in many contexts.

Predictive Validity of the Dynamic Risk Assessment for Offender Re-Entry Among Intimate Partner Violence Offenders

“…deviations from a 50% base rate) on dichotomous outcome variables, such as the endorsement rate on the proportion of a subgroup in a sample (e.g., the proportion of Aboriginal versus non-Aboriginal offenders; Babchishin & Helmus, 2016). This is important for the current research question, examined differences in DRAOR Total, domain, and item scores between 112…”

“…The null findings for general and violent recidivism could be due to the small number of recidivists (n general = 32, n violent = 11). While AUCs are robust to attenuations in magnitude due to high or low base rates (Babchishin & Helmus, 2016), 100 events and 100 nonevents are recommended for adequate statistical power when predicting dichotomous outcomes in logistic regression (Vergouwe, Steyerberg, Eijkemans, & Habbema, 2005) Additional work by Yesberg and colleagues (2015) evaluated whether the DRAOR predicted general recidivism for both male (n = 133) and female (n = 133) parolees in New Zealand, and whether there were differences in predictive accuracy between genders. This sample is a subset of the dataset used by Hanby (2013), representing all offender types.…”

“…The AUC is an effect size statistic that is appropriate when one variable is dichotomous and the other is dichotomous, ordinal, or interval (Swets, Dawes, & Monahan 2000). AUCs are one of the most commonly recommended effect size statistics for examining relationships to recidivism (Helmus & Babchishin, 2017), but are rarely used for dichotomous variables other than recidivism (Babchishin & Helmus, 2016). Recent research, for example, has utilized AUCs in examining differences between Aboriginal and non-Aboriginal offenders Perley-Robertson, Helmus, & Forth, 2018).…”

“…An AUC in this context is interpreted as the "probability that a randomly selected Aboriginal offender will have a higher risk score than a randomly selected non-Aboriginal offender" (Babchishin & Helmus, 2016, p. 1023.…”

Telltale Gait: The Role of Nonverbal Cues in Perceptions of Personality and Vulnerability

“…A series of Area Under the Receiver Operating Characteristic Curve (AUC) analyses were first conducted to determine whether the display of vulnerable gait cues in the unaware and aware conditions were associated with victimization history, and whether this association was dependent on the type of victimization experienced by the walker (i.e., any, exclusively sexual, and exclusively violent; see Table 6). AUC analyses are recommended over traditional correlational analyses when one of the variables is dichotomous (i.e., victimization history; Babchishin & Helmus, 2015;Rice & Harris, 2005). An AUC of .56 reflects a small effect size, and AUC of .64 reflects a moderate effect size, and an AUC of .71 reflects a large effect size (Rice & Harris, 2005).…”