Statistical Significance Testing at CHI PLAY: Challenges and Opportunities for More Transparency

Abstract

Statistical Significance Testing -- or Null Hypothesis Significance Testing (NHST) -- is common to quantitative CHI PLAY research. Drawing from recent work in HCI and psychology promoting transparent statistics and the reduction of questionable research practices, we systematically review the reporting quality of 119 CHI PLAY papers using NHST (data and analysis plan at https://osf.io/4mcbn/. We find that over half of these papers employ NHST without specific statistical hypotheses or research questions, which may risk the proliferation of false positive findings. Moreover, we observe inconsistencies in the reporting of sample sizes and statistical tests. These issues reflect fundamental incompatibilities between NHST and the frequently exploratory work common to CHI PLAY. We discuss the complementary roles of exploratory and confirmatory research, and provide a template for more transparent research and reporting practices.

Are You Open? A Content Analysis of Transparency and Openness Guidelines in HCI Journals

“…Thus, Hornbaek et al [25] found that only 3% of HCI articles across 4 outlets in 2014 were replications, and many of these accidental. Vornhagen and colleagues [53] investigated current practices involving transparency and null hypothesis significance testing in papers at the CHI sister conference CHI PLAY. Only 20% of the studied CHI PLAY papers included detailed materials (questionnaires, software, etc), less than 5% were accompanied by open data, and none of them shared their analysis code.…”