Designing for Preregistration

Pu, Xiaoying; Zhu, Licheng; Kay, Matthew; Conrad, Frederick G.

doi:10.1145/3290607.3312862

Cited by 6 publications

(5 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Pre-registration [4,33,117,131,170,177]: The study is pre-registered at a permanent third-party archive (e.g., OSF.io or AsPredicted.org) and accompanied by timestamps. Johanson et al [81], for instance, provide an example of a pre-registration plan in the context of player-computer interaction.…”

Section: Pre-registrationmentioning

confidence: 99%

“…• Deviations from pre-registration are justifed [33,117,131,170,177]: Studies do not always go as intended. Deviations from the pre-registration plan are often warranted, but should be clearly highlighted and justifed in the paper.…”

Section: Pre-registrationmentioning

confidence: 99%

“…These Questionable Research Practices [177,QRPs] threaten the legitimacy of statistical signifcance and therefore complicate interpretation of published research fndings [79,154]. QRPs are facilitated by a publishing climate biased towards statistically signifcant results 1 , leaving non-signifcant research fndings in the fle drawer [33,49,131,170].…”

mentioning

confidence: 99%

See 2 more Smart Citations

Statistical Significance Testing at CHI PLAY: Challenges and Opportunities for More Transparency

Vornhagen

Tyack

Mekler

2020

Proceedings of the Annual Symposium on Computer-Human Interaction in Play

View full text Add to dashboard Cite

Statistical Signifcance Testing -or Null Hypothesis Signifcance Testing (NHST) -is common to quantitative CHI PLAY research. Drawing from recent work in HCI and psychology promoting transparent statistics and the reduction of questionable research practices, we systematically review the reporting quality of 119 CHI PLAY papers using NHST (data and analysis plan at OSF.io). We fnd that over half of these papers employ NHST without specifc statistical hypotheses or research questions, which may risk the proliferation of false positive fndings. Moreover, we observe inconsistencies in the reporting of sample sizes and statistical tests. These issues refect fundamental incompatibilities between NHST and the frequently exploratory work common to CHI PLAY. We discuss the complementary roles of exploratory and confrmatory research, and provide a template for more transparent research and reporting practices. CCS CONCEPTS• Human-centered computing → Empirical studies in HCI .

show abstract

Section: Pre-registrationmentioning

confidence: 99%

Section: Pre-registrationmentioning

confidence: 99%

See 1 more Smart Citation

Statistical Significance Testing at CHI PLAY: Challenges and Opportunities for More Transparency

Vornhagen

Tyack

Mekler

2020

Proceedings of the Annual Symposium on Computer-Human Interaction in Play

View full text Add to dashboard Cite

show abstract

“…Notable community efforts include the RepliCHI workshop series [58] deliberating the form, fit, and value of replications for HCI research, recently reiterated by Cockburn and colleagues [8], and the CHI Transparent Statistics events [28], which have resulted in recommendations for revising the author submission guidelines of CHI (https://transparentstatistics.org/). A good number of HCI and CSCW researchers have been looking into socio-technical, design, and usability barriers and solutions to the adoption of open science practices and tools [15,16,32,44,45,52]. Others have empirically assessed the status of openness and transparency in the Figure 1: Hierarchy of research culture factors affecting the adoption of openness and transparency standards, with focus of current paper highlighted in blue.…”

Section: Open Science and Hcimentioning

confidence: 99%

Are You Open? A Content Analysis of Transparency and Openness Guidelines in HCI Journals

Ballou

Warriar

Deterding

2021

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Within the wider open science reform movement, HCI researchers are actively debating how to foster transparency in their own field. Publication venues play a crucial role in instituting open science practices, especially journals, whose procedures arguably lend themselves better to them than conferences. Yet we know little about how much HCI journals presently support open science practices. We identified the 51 most frequently published-in journals by recent CHI first authors and coded them according to the Transparency and Openness Promotion guidelines, a high-profile standard of evaluating editorial practices. Results indicate that journals in our sample currently do not set or specify clear openness and transparency standards. Out of a maximum of 29, the modal score was 0 (mean = 2.5, SD = 3.6, max = 15). We discuss potential reasons, the aptness of natural science-based guidelines for HCI, and next steps for the HCI community in furthering openness and transparency. CCS CONCEPTS• Human-centered computing → HCI theory, concepts and models; Empirical studies in HCI.

show abstract

“…These studies are devalued as unoriginal and not contributing anything beyond what is already known. This pattern led to a replication crisis in fields including psychology, HCI, physics, and economics [5,23,25,48,69,71,80] in which the community fails to independently replicate prior studies in their fields. Failure to address this crisis means that a field cannot trust its theories, methods, and basic assumptions.…”

Section: Introductionmentioning

confidence: 99%

More Useful Junk? Replicating the Effects of Visual Embellishment on Description and Recall of Charts

Syeda¹,

South²,

Raynor³

et al. 2023

Preprint

View full text Add to dashboard Cite

Data visualization is a young and growing field with many unresolved, debated, and untested theories that need to be tested, validated, and strengthened. Replication studies are a common way to achieve this validation by investigating the credibility, rigor, and generalizability of previously published research. The execution of replication studies is crucial to strengthen the field of data visualization and to ensure that the field’s foundations are solid with accurate methods and theories. Although visualization researchers acknowledge the epistemological significance of replications and its necessity to establish trust and reliability, the field has made little progress to support the publication of such studies and importantly provide methods to the community to encourage replications. In this paper, we contribute Vis Repligogy, a novel framework to incorporate replication studies within visualization course curricula that not only teaches students replication and evaluation methodologies but also results in executed replication studies to validate prior work. To validate the feasibility of the framework, we present the results of four replication studies that were conducted in a graduate data visualization course that implemented the Vis Repligogy framework. We also provide a meta-analysis of the four replication studies that replicated and validated, a seminal work in data visualization investigating the effects of visual embellishments on chart interpretability and memorability. Finally, we reflect on our experience of conducting the replication studies through the Vis Repligogy framework and reflect on the experience to provide useful recommendations for future use of the framework. We envision that this framework will encourage instructors to conduct replications in their own courses and will help facilitate more replication studies in the visualization community to support a culture shift where replications are commonplace.

show abstract

Designing for Preregistration

Cited by 6 publications

References 5 publications

Statistical Significance Testing at CHI PLAY: Challenges and Opportunities for More Transparency

Statistical Significance Testing at CHI PLAY: Challenges and Opportunities for More Transparency

Are You Open? A Content Analysis of Transparency and Openness Guidelines in HCI Journals

More Useful Junk? Replicating the Effects of Visual Embellishment on Description and Recall of Charts

Contact Info

Product

Resources

About