Making Science Transparent By Default; Introducing the TOP Statement

Abstract

In order to increase the replicability of scientific work, the scientific community has called for practices designed to increase the transparency of research (McNutt, 2014; Nosek et al., 2015). The validity of a scientific claim depends not on the reputation of those making the claim, the venue in which the claim is made, or the novelty of the result, but rather on the empirical evidence provided by the underlying data and methods. Proper evaluation of the merits of scientific findings requires availability of the methods, materials, and data and the reasoned argument that serve as the basis for the published conclusions (Claerbout and Karrenbach 1992; Donoho et al 2009; Stodden et al 2013; Borwein et al 2013; Munafò et al, 2017). Wide and growing support for these principles (see, for example, signatories to Declaration on Research Assessment, DORA, https://sfdora.org/, and the Transparency and Openness Promotion Guidelines https://cos.io/our-services/top-guidelines/) must be coupled with guidelines to increase open sharing of data and research materials, use of reporting guidelines, preregistration, and replication. We propose that, going forward, authors of all scientific articles disclose the availability and location of all research items, including data, materials, and code, related to their published articles in what we will refer to as a TOP Statement.

A consensus-based transparency checklist

“…Online applications for the checklist comment their adherence to these TOP standards by adding a transparency statement in their articles (TOP Statement) 9 . Collectively, these somewhat piecemeal innovations illustrate a science-wide shift toward greater transparency in research reports.…”

Does preregistration improve the credibility of research findings?

“…However, historical transparency is not necessary when contemporary transparency is available. For example, researchers can confirm the absence of p-hacking in their research reports by (a) actively affirming the disclosure of their data collection stopping rule, data exclusions, measures, and manipulations (Simmons, Nelson, & Simonsohn, 2012), (b) providing logical and principled justifications for nonstandard data exclusions and analytical approaches (Giner-Sorolla, 2012, p. 568), (c) providing public access to their standard data analysis procedures (e.g., Lin & Green, 2016), (d) providing public access to their research materi-als, data, and coding information (e.g., Aalbersberg et al, 2018), and (e) reporting the results of robustness analyses (e.g., Thabane et al, 2013). Identifying Optional Stopping.…”

Does preregistration improve the credibility of research findings?

“…However, historical transparency is not necessary when contemporary transparency is available. For example, researchers can confirm the absence of p-hacking in their research reports by (a) actively affirming the disclosure of their data collection stopping rule, data exclusions, measures, and manipulations (Simmons et al, 2012), (b) providing logical and principled justifications for nonstandard data exclusions and analytical approaches (Giner-Sorolla, 2012, p. 568), (c) providing public access to their standard data analysis procedures (e.g., Lin & Green, 2016), (d) providing public access to their research materials, data, and coding information (e.g., Aalbersberg et al, 2018), and (e) reporting the results of robustness analyses (e.g., Thabane et al, 2013).…”