Moving Transparent Statistics Forward at CHI

Abstract

Transparent statistics is a philosophy of statistical reporting whose purpose is scientific advancement rather than persuasion. We ran a SIG at CHI 2016 to discuss problems and limitations in statistical practices in HCI and options for moving the field towards clearer and more reliable ways of writing about experiments, and received an overwhelming response. This SIG resulted in rough drafts of reviewer guidelines, resources for authors, and other suggestions for advancing a vision of transparent statistics within the field; this year, we propose a concentrated one-day writing workshop to develop those documents into a polished state with input from a diverse cross-section of the CHI community. Moving Transparent Statistics Forward at CHI AbstractTransparent statistics is a philosophy of statistical reporting whose purpose is scientific advancement rather than persuasion. We ran a SIG at CHI 2016 to discuss problems and limitations in statistical practices in HCI and options for moving the field towards clearer and more reliable ways of writing about experiments, and received an overwhelming response. This SIG resulted in rough drafts of reviewer guidelines, resources for authors, and other suggestions for advancing a vision of transparent statistics within the field; this year, we propose a concentrated one-day writing workshop to develop those documents into a polished state with input from a diverse cross-section of the CHI community.

Transparency of CHI Research Artifacts: Results of a Self-Reported Survey

“…In a Special Interest Group (SIG) meeting at CHI 2018, Chuang & Pfeil lead a discussion on how the TOP guidelines match current practices at CHI [7]. Another SIG meeting, focusing on transparent statistics, was also held at CHI 2018 [50], following related workshops/SIGs in previous years [25,26]. Echtler & Häußler analyzed papers from CHI 2016-17 and found that less than 5% provided any kind of publicly available source code [10].…”

Increasing the Transparency of Research Papers with Explorable Multiverse Analyses

“…Much of this movement calls for more transparency in the way statistics are reported, so that findings become more trustworthy, more likely to be interpreted correctly, and easier to verify and replicate [29,69,72]. Concern for transparency in statistical reporting has spread to the HCI community, which has published several articles [24,31,58] and hosted several workshops [56,57,96] on the topic.…”

Open is not enough

“…The Transparent Statistics in HCI Working Group 13 concluded a Special Interest Group (SIG) at CHI 2016 [104], a workshop at CHI 2017 [105], and a SIG at CHI 2018 [193]. At the time of writing, the working group proposed nine guiding principles 14 .…”