Visualization strategies for regression estimates with randomization inference

Abstract

Coefficient plots are a popular tool for visualizing regression estimates. The appeal of these plots is that they visualize confidence intervals around the estimates and generally center the plot around zero, meaning that any estimate that crosses zero is statistically nonsignificant at least at the alpha level around which the confidence intervals are constructed. For models with statistical significance levels determined via randomization models of inference and for which there is no standard error or confidence intervals for the estimate itself, these plots appear less useful. In this article, I illustrate a variant of the coefficient plot for regression models with p-values constructed using permutation tests. These visualizations plot each estimate’s p-value and its associated confidence interval in relation to a specified alpha level. These plots can help the analyst interpret and report the statistical and substantive significances of their models. I illustrate using a nonprobability sample of activists and participants at a 1962 anticommunism school.

Real‐time irrigation diversion data delivery can benefit adaptive capacity in communal irrigation systems

“…Permutation tests are most ideal in the context of random assignment in experimental designs. This is not the case for the present study; however, as Manly notes (2007, p. 180), permutation tests with a data generating mechanism might also be used when neither random sampling nor random assignment can be assumed (see also Taylor, 2020, p. 312). For each paired t test, we first calculated the observed t statistic, randomly sampled one possible permutation of the pre‐test adaptive capacity variable, ran the paired t test again, and then calculated the simulated t statistic.…”

“…Due to our snowball sampling survey data collection approach (i.e., not random sampling), we adopted a randomization inference model (Ludbrook & Dudley, 1998) and reported paired sample t tests with Monte Carlo permutation tests (Ernst, 2004; Manly, 2007; Taylor, 2020). Implementing paired sample t tests with asymptotically derived standard errors and population inference with these survey data would violate the random sampling assumption.…”

“…Due to our snowball sampling survey data collection approach (i.e., not random sampling), we adopted a randomization inference model (Ludbrook & Dudley, 1998) and reported paired sample t tests with Monte Carlo permutation tests (Ernst, 2004;Manly, 2007;Taylor, 2020).…”