Validation of Visual Statistical Inference, Applied to Linear Models

Abstract

Statistical graphics play a crucial role in exploratory data analysis, model checking, and diagnosis. The lineup protocol enables statistical significance testing of visual findings, bridging the gulf between exploratory and inferential statistics. In this article, inferential methods for statistical graphics are developed further by refining the terminology of visual inference and framing the lineup protocol in a context that allows direct comparison with conventional tests in scenarios when a conventional test exists. This framework is used to compare the performance of the lineup protocol against conventional statistical testing in the scenario of fitting linear models. A human subjects experiment is conducted using simulated data to provide controlled conditions. Results suggest that the lineup protocol performs comparably with the conventional tests, and expectedly outperforms them when data are contaminated, a scenario where assumptions required for performing a conventional test are violated. Surprisingly, visual tests have higher power than the conventional tests when the effect size is large. And, interestingly, there may be some super-visual individuals who yield better performance and power than the conventional test even in the most difficult tasks. Supplementary materials for this article are available online.

Graphical Inference in Geographical Research

“…where X has a binomial distribution with parameters K (number of judges) and P (probability of random selection 5 1/m) (Majumder, Hofmann, and Cook 2013). Therefore, if a single judge is able to identify the real data in a line-up involving 19 decoys (P 5 1/20), then the P-value is 0.05 (Buja et al 2009).…”

“…Power is often used to choose an appropriate plot design or test but it can also be used to compare the performance of visual and traditional tests (Hofmann et al 2012;Majumder, Hofmann, and Cook 2013). Power is often used to choose an appropriate plot design or test but it can also be used to compare the performance of visual and traditional tests (Hofmann et al 2012;Majumder, Hofmann, and Cook 2013).…”

“…It also increases false positive results in scientific literature due to the pressure to publish only significant findings (Head et al 2015). This method bridges the gap between exploratory and confirmatory analysis and allows for direct comparison with traditional statistical tests (Majumder, Hofmann, and Cook 2013). By requiring both thoughtful consideration of the null hypothesis as well as an unbiased jury, graphical inference makes manipulation difficult and thereby limits the opportunity for these problems to arise.…”

“…Here, we focus on the latter study where we use visualizations to conduct hypothesis testing through graphical inference. This method bridges the gap between exploratory and confirmatory analysis and allows for direct comparison with traditional statistical tests (Majumder, Hofmann, and Cook 2013).…”

Discussion of “visualizing statistical models: Removing the blindfold”

“…We have called this approach to understanding data analyst behavior "evidence based data analysis" (EBDA) [6] and this type of experiment has a rich tradition in the data visualization community [1,2]. We would be eager to hear how the authors propose to combine the visualizations described in this work with EBDA given the leadership role the authors have played in EBDA for data visualization [7,8]. From the NewHavenResidential dataset in the barcode package, which provides information about housing prices and attributes.…”