Transparency of CHI Research Artifacts: Results of a Self-Reported Survey

Abstract

Several fields of science are experiencing a ""replication crisis"" that has negatively impacted their credibility. Assessing the validity of a contribution via replicability of its experimental evidence and reproducibility of its analyses requires access to relevant study materials, data, and code. Failing to share them limits the ability to scrutinize or build-upon the research, ultimately hindering scientific progress.Understanding how the diverse research artifacts in HCI impact sharing can help produce informed recommendations for individual researchers and policy-makers in HCI. Therefore, we surveyed authors of CHI 2018-2019 papers, asking if they share their papers' research materials and data, how they share them, and why they do not. The results (34% response rate) show that sharing is uncommon, partly due to misunderstandings about the purpose of sharing and reliable hosting. We conclude with recommendations for fostering open research practices.This paper and all data and materials are freely available at https://osf.io/3bu6t.

Are You Open? A Content Analysis of Transparency and Openness Guidelines in HCI Journals

“…Recognition of the above problems and calls to action have permeated human-computer interaction (HCI) [8,12,55,58] and connected fields, including computer science [7], health informatics [9], graphics and visualization [4,30], and computing education [1].…”

A Meta-Analysis of Effect Sizes of CHI Typing Experiments

“…From eight missing responses, seven papers reported partial data, which are included in this meta-analysis. In a survey [41], the authors have found that some researchers are not willing to share the data and information. It could be the case that some of the contacted authors had a similar opinion.…”

“…I acknowledge that my work has the following three limitations. First, since there is no common practice of preregistration in the HCI community, it is almost impossible to analyze the results of unpublished studies [4,41]. Therefore, I could not perform the publication bias assessment.…”

Community expectations for research artifacts and evaluation processes

“…Wacharamanotham et al inspected the low availability of artifacts in the HCI community and found that four factors influence researchers to refrain from sharing artifacts: concern about personally-identifiable data, lack of participant's permission, lack of motivation, resources, or recognition, and doubt in the usefulness of their artifact outside their own study [24]. Dahlgren conducted an observatory study during the OOPSLA 2019 artifact evaluation and found that the most prominent negative comments during artifact review are due to limited physical resources or review time to test artifacts and problems with documentation [8].…”

“…Contrary to the ACM guidelines, 61 out of 79 analyzed calls for artifacts explicitly state a purpose for artifact evaluation. Across all analyzed calls, the most frequently named purpose of artifact evaluation processes is to enable reuse of artifacts (32 calls 6 ), followed by reproducibility (24) and enabling comparison (17) for future research against published results. When divided by community, programming languages conferences 7 named reproducibility (21) more often than reuse.…”