Increasing the Transparency of Research Papers with Explorable Multiverse Analyses

Dragicevic, Pierre; Jansen, Yvonne; Sarma, Abhraneel; Kay, Matthew; Chevalier, Fanny

doi:10.1145/3290605.3300295

Cited by 110 publications

(124 citation statements)

References 68 publications

Supporting

Mentioning

110

Contrasting

Unclassified

Order By: Relevance

“…The third strategy has already been performed. Robustness analyses were included for Parts 2 and 3 to minimize researcher degrees of freedom (Simmons et al, 2011) by providing a (simplified) type of multiverse analysis of the results (see Dragicevic et al, 2019;Steegan et al, 2016). These are discussed in more detail above.…”

Section: Limitations Of the Studymentioning

confidence: 99%

Appraising Contemporary Social Work Research: Meta-Research on Statistical Reporting, Statistical Power, and Evidential Value

Dunleavy¹

2020

Preprint

View full text Add to dashboard Cite

Background. In recent years, the veracity of scientific findings has come under intense scrutinyin what has been called the “replication crisis” (sometimes called the “reproducibility crisis” or“crisis of confidence”). This crisis is marked by the propagation of scientific claims which weresubsequently contested, found to be exaggerated, or deemed false. The causes of this crisis aremany, but include poor research design, inappropriate statistical analysis, and the manipulationof study results. Though it is uncertain if social work is in the midst of a similar crisis, it is notunlikely, given parallels between the field and adjacent disciplines in crisis.Objective. This dissertation aims to articulate these problems, as well as foundational issues instatistical theory, in order to scrutinize statistical practice in social work research. In doing so, itparallels recent work in psychology, neuroscience, medicine, ecology, and other scientificdisciplines, while introducing a new program of meta-research to the social work profession.Method. Five leading social work journals were analyzed across a five-year period (2014-2018).In all 1,906 articles were reviewed, with 310 meeting inclusion criteria. The study was dividedinto three complementary parts. Statistical reporting practices were coded and analyzed in Part 1of the study (n = 310). Using reported sample sizes from these articles, a power survey wasperformed, in Part 2, for small, medium, and large effect sizes (n = 207). A novel statistical tool,the p-curve, was used in Part 3 to evaluate the evidential value of results from one journal(Research on Social Work Practice) and to assess for bias. Results from 39 of the 78 eligiblearticles were included in the analysis. Data and materials are available at: https://osf.io/45z3h/Results. Part 1: Notably, 86.1% of articles reviewed did not report an explicit alpha level. Apower analysis was performed in only 7.4% of articles. Use of p-values was common, beingreported in 96.8% of articles, but only 29% of articles reported them in exact form. Only 36.5%of articles reported confidence intervals; with the 95% coverage rate being the most common(reported in 31.3% of all studies). Effect sizes were explicitly reported in the results section ortables in a little more than half of articles (55.2%). Part 2: The mean statistical power for articleswas 57% for small effects, 88% for medium effects, and 95% for large effects. 61% of studiesdid not have adequate power (.80) to detect a small effect, 19% did not have adequate power todetect a medium effect, and 7% a large effect. A robustness test yielded similar but moreconservative estimates for these findings. Part 3: Both the primary p-curve and robustness testyielded right-skewed curves, indicating evidential value for the included set of results, and noevidence of bias.Conclusion. Overall, these findings provide a snapshot of the status of contemporary social workresearch. The results are preliminary but indicate areas where statistical design and reporting canbe improved in published research. The results of the power survey suggest that the field hasincreased mean statistical power compared to prior decades; though these findings are tentativeand have numerous limitations. The results of the p-curve demonstrate its potential as a tool forinvestigating bias within published research; while suggesting that the results included fromResearch on Social Work Practice have evidential value. In all this study provides a first steptowards a broader and more comprehensive assessment of the field.

show abstract

Section: Limitations Of the Studymentioning

confidence: 99%

Appraising Contemporary Social Work Research: Meta-Research on Statistical Reporting, Statistical Power, and Evidential Value

Dunleavy¹

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Closely related are Dragicevic et al's [27] explorable multiverse analysis reports (EMAR) which introduce interactive graphics into the results sections of research articles. In contrast to this work, we focus on interactions specifically for multiple small-scale visualizations and not on in-text controls for modifying the parameters of larger visualizations.…”

Section: Interactive Text Analysismentioning

confidence: 99%

Interaction Techniques for Visual Exploration Using Embedded Word-Scale Visualizations

Goffin

Blascheck

Isenberg

et al. 2020

Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

We describe a design space of view manipulation interactions for small data-driven contextual visualizations (word-scale visualizations). These interaction techniques support an active reading experience and engage readers through exploration of embedded visualizations whose placement and content connect them to specific terms in a document. A reader could, for example, use our proposed interaction techniques to explore word-scale visualizations of stock market trends for companies listed in a market overview article. When readers wish to engage more deeply with the data, they can collect, arrange, compare, and navigate the document using the embedded word-scale visualizations, permitting more visualizationcentric analyses. We support our design space with a concrete implementation, illustrate it with examples from three application domains, and report results from two experiments. The experiments show how view manipulation interactions helped readers examine embedded visualizations more quickly and with less scrolling and yielded qualitative feedback on usability and future opportunities.

show abstract

“…Victor designed the Website worrydream, which allows readers to change the assumptions, i.e. the initial configuration of the parameters, underlying the results in the text [49] (see also the work by Dragicevic et al [11]). Based on those changes, texts and figures adapt accordingly.…”

Section: Creating Interactive Papersmentioning

confidence: 99%

“…One of the key motivations of this work on interactive papers is the possibility to gain an increased understanding. Although some studies and related approaches provided initial evidence that interactive papers are better understandable and more transparent (see, for example, the work by Adnan et al [1], Dragicevic et al [11], and Wacharamanotham et al [50]), it remains open to which extent readers obtain additional knowledge compared to 'classic' papers, such as PDFs. Further studies with authors and readers are required to evaluate the bindings concept, for example, in comparison to the existing approaches presented in Related Work.…”

Section: Limitationsmentioning

confidence: 99%

Creating Interactive Scientific Publications using Bindings

Konkol

Kray

Suleiman

2019

Proc. ACM Hum.-Comput. Interact.

View full text Add to dashboard Cite

Many scientific publications report on computational results based on code and data, but even when code and data are published, the main text is usually provided in a separate, traditional format such as PDF. Since code, data, and text are not linked on a deep level, it is difficult for readers and reviewers to understand and retrace how the authors achieved a specific result that is reported in the main text, e.g. a figure, table, or number. In addition, to make use of new the opportunities afforded by data and code availability, such as re-running analyses with changed parameters, considerable effort is required. In order to overcome this issue and to enable more interactive publications that support scientists in more deeply exploring the reported results, we present the concept, implementation, and initial evaluation of bindings. A binding describes which data subsets, code lines, and parameters produce a specific result that is reported in the main text (e.g. as a figure, table, or number). Based on a prototypical implementation of these bindings, we propose a toolkit for authors to easily create interactive figures by connecting specific UI widgets (e.g. a slider) to parameters. In addition to inspecting code and data, readers can then manipulate the parameter and see how the results change. We evaluated the approach by applying it to a set of existing articles. The results provide initial evidence that the concept is feasible and applicable to many papers with moderate effort.

show abstract

Increasing the Transparency of Research Papers with Explorable Multiverse Analyses

Cited by 110 publications

References 68 publications

Appraising Contemporary Social Work Research: Meta-Research on Statistical Reporting, Statistical Power, and Evidential Value

Appraising Contemporary Social Work Research: Meta-Research on Statistical Reporting, Statistical Power, and Evidential Value

Interaction Techniques for Visual Exploration Using Embedded Word-Scale Visualizations

Creating Interactive Scientific Publications using Bindings

Contact Info

Product

Resources

About