Transparent Practices for Quantitative Empirical Research

Wacharamanotham, Chat; Yang, Fumeng; Pu, Xiaoying; Sarma, Abhraneel; Padilla, Lace

doi:10.1145/3491101.3503760

Cited by 7 publications

(3 citation statements)

References 42 publications

(34 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, statistical procedures like multiverse analysis [52,54] have been proposed as part of the movement of increasing transparency in research practices, which also calls for sharing research materials, data, analysis scripts etc. [58,59].…”

Section: Challenges To Adoption Of Multiverse Visualisation Interfacesmentioning

confidence: 99%

Milliways: Taming Multiverses through Principled Evaluation of Data Analysis Paths

Sarma,

Hwang,

Hullman

et al. 2024

Preprint

View full text Add to dashboard Cite

Multiverse analyses involve conducting all combinations of reasonable choices in a data analysis process. A reader of a study containing a multiverse analysis might question—are all the choices included in the multiverse reasonable and equally justifiable? How much do results vary if we make different choices in the analysis process? In this work, we identify principles for validating the composition of, and interpreting the uncertainty in, the results of a multiverse analysis. We present Milliways, a novel interactive visualisation system to support principled evaluation of multiverse analyses. Milliways provides interlinked panels presenting result distributions, individual analysis composition, multiverse code specification, and data summaries. Milliways supports interactions to sort, filter and aggregate results based on the analysis specification to identify decisions in the analysis process to which the results are sensitive. To represent the two qualitatively different types of uncertainty that arise in multiverse analyses—probabilistic uncertainty from estimating unknown quantities of interest such as regression coefficients, and possibilistic uncertainty from choices in the data analysis—Milliways uses consonance curves and probability boxes. Through an evaluative study with five users familiar with multiverse analysis, we demonstrate how Milliways can support multiverse analysis tasks, including a principled assessment of the results of a multiverse analysis.

show abstract

Section: Challenges To Adoption Of Multiverse Visualisation Interfacesmentioning

confidence: 99%

Milliways: Taming Multiverses through Principled Evaluation of Data Analysis Paths

Sarma,

Hwang,

Hullman

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…We recommend that HCI venues provide specific guidelines with detailed instructions on how to meet each practice. Some of these practices require special skills and training, such as transparency practices for quantitative studies [96]. The current CHI guidelines somewhat support transparency, but they should also raise awareness about research ethics and openness.…”

Section: Raising Awarenessmentioning

confidence: 99%

Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022

Niksirat¹,

Goswami²,

Rao³

et al. 2023

Preprint

View full text Add to dashboard Cite

In recent years, various initiatives from within and outside the HCI field have encouraged researchers to improve research ethics, openness, and transparency in their empirical research. We quantify how the CHI literature might have changed in these three aspects by analyzing samples of 118 CHI 2017 and 127 CHI 2022 papers—randomly drawn and stratified across conference sessions. We operationalized research ethics, openness, and transparency into 45 criteria and manually annotated the sampled papers. The results show that the CHI 2022 sample was better in 18 criteria, but in the rest of the criteria, it has no improvement. The most noticeable improvements were related to research transparency (10 out of 17 criteria). We also explored the possibility of assisting the verification process by developing a proof-of-concept screening system. We tested this tool with eight criteria. Six of them achieved high accuracy and F1 score. We discuss the implications for future research practices and education.

show abstract

“…In response to threats to the validity of empirical studies popularized through events like the so-called "replication crisis" occurring in social science, human-computer interaction and visualization researchers have advocated for alternatives to Null Hypothesis Significance test-ing [20,21], advocated for pre-registration [22], produced guidelines for transparent statistical reporting [23], Such approaches are very well-motivated to increase the transparency of visualization research. However, they do not necessarily address the interpretability of experiment results, because they are largely orthogonal to questions like what task to study in an experiment.…”

Section: Visualization Evaluationmentioning

confidence: 99%

The Rational Agent Benchmark for Data Visualization

Wu¹,

ZiyangGuo²,

Mamakos³

et al. 2023

Preprint

View full text Add to dashboard Cite

Understanding how helpful a visualization is from experimental results is difficult because the observed performance is confounded with aspects of the study design, such as how useful the information that is visualized is for the task. We develop a rational agent framework for designing and interpreting visualization experiments. Our framework conceives two experiments with the same setup: one with behavioral agents (human subjects), the other one with a hypothetical rational agent. A visualization is evaluated by comparing the expected performance of behavioral agents to that of rational agent under different assumptions. Using recent visualization decision studies from the literature, we demonstrate how the framework can be used to pre-experimentally evaluate the experiment design by bounding the expected improvement in performance from having access to visualizations, and post-experimentally to deconfound errors of information extraction from errors of optimization, among other analyses.

show abstract

Transparent Practices for Quantitative Empirical Research

Cited by 7 publications

References 42 publications

Milliways: Taming Multiverses through Principled Evaluation of Data Analysis Paths

Milliways: Taming Multiverses through Principled Evaluation of Data Analysis Paths

Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022

The Rational Agent Benchmark for Data Visualization

Contact Info

Product

Resources

About