Share the code, not just the data: A case study of the reproducibility of articles published in the Journal of Memory and Language under the open data policy

Laurinavichyute, Anna; Yadav, Himanshu; Vasishth, Shravan

doi:10.1016/j.jml.2022.104332

Cited by 22 publications

(25 citation statements)

References 172 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Lack of data access is another key challenge to reproducibility. Adoption of open-science practices, including sharing of data and code, represents a partial solution to this issue [ 57 , 58 ], acknowledging that not all data can be shared openly owing to privacy concerns. Similarly, transparent descriptions of data collection and analytic methods are necessary for reproduction [ 59 ].…”

Section: Approaches To Improving Reproducibility and Scientific Integ...mentioning

confidence: 99%

“…For example, the Journal of Memory and Language adopted a mandatory data-sharing policy in 2019. An evaluation of this policy implementation found that data sharing increased more than 50% and the strongest predictor for reproducibility was the sharing of analysis code, increasing the probability of reproducibility by 40% [ 57 ]. Such practices are also fostered by the creation and use of infrastructure, such as the aforementioned SoBigData, and reproducibility network groups, such as the Swiss Reproducibility Network, a peer-lead group that aims to improve both replicability and reproducibility [ 74 ], improve communication, collaboration, and encourage the use of rigorous research practices.…”

Section: Approaches To Improving Reproducibility and Scientific Integ...mentioning

confidence: 99%

See 1 more Smart Citation

Reproducibility and Scientific Integrity of Big Data Research in Urban Public Health and Digital Epidemiology: A Call to Action

Gutierrez

Lindegger

Heravi

et al. 2023

IJERPH

View full text Add to dashboard Cite

The emergence of big data science presents a unique opportunity to improve public-health research practices. Because working with big data is inherently complex, big data research must be clear and transparent to avoid reproducibility issues and positively impact population health. Timely implementation of solution-focused approaches is critical as new data sources and methods take root in public-health research, including urban public health and digital epidemiology. This commentary highlights methodological and analytic approaches that can reduce research waste and improve the reproducibility and replicability of big data research in public health. The recommendations described in this commentary, including a focus on practices, publication norms, and education, are neither exhaustive nor unique to big data, but, nonetheless, implementing them can broadly improve public-health research. Clearly defined and openly shared guidelines will not only improve the quality of current research practices but also initiate change at multiple levels: the individual level, the institutional level, and the international level.

show abstract

Section: Approaches To Improving Reproducibility and Scientific Integ...mentioning

confidence: 99%

Section: Approaches To Improving Reproducibility and Scientific Integ...mentioning

confidence: 99%

Reproducibility and Scientific Integrity of Big Data Research in Urban Public Health and Digital Epidemiology: A Call to Action

Gutierrez

Lindegger

Heravi

et al. 2023

IJERPH

View full text Add to dashboard Cite

show abstract

“…raw data and processing data) as well as the code-syntax that was used to carry out the analyses in the given software 10 . Publishing the code-syntax is useful for promoting reproducibility, as it allows the same analytical steps to be followed, but also promotes quality control through increased opportunities to find bugs in the code (Laurinavichyute et al, 2022). All of this is especially 10 One of the reviewers suggested indicating, and we agreed with him, the need to provide open access also to the total output file produced by software programmes such as M-plus or the Lavaan package in R. As the reviewer says, "the total output file is very informative for other researchers in order to replicate the reported results".…”

Section: Be Open Sciencementioning

confidence: 99%

How do you Behave as a Psychometrician? Research Conduct in the Context of Psychometric Research

Flores-Kanter¹,

Mosquera²

2023

Preprint

View full text Add to dashboard Cite

The identification of fraudulent and questionable research conduct is not something new. However, in the last 12 years the aim has been to identify specific problems and concrete solutions applicable to each area of knowledge. For example, previous work has focused on questionable and responsible research conducts associated with clinical assessment, measurement practices in psychology and related sciences; or applicable to specific areas of study, such as suicidology. One area of study that merits further study of questionable and responsible research behaviors is psychometrics. Focusing on psychometric research is important and necessary, as without adequate evidence of construct validity the overall validity of the research is at least debatable. Our interest here is to (a) identifying questionable research conduct specifically linked to psychometric studies; and (b) promoting greater awareness and widespread application of responsible research conduct in psychometrics research. We believe that the identification and recognition of these conducts is important and will help us to improve our daily work as psychometricians.

show abstract

“…A few examples illustrate the reality. Laurinavichyute et al (2022) were able to reproduce the results of only 34% to 56% of the 59 papers that they chose to inspect from the Journal of Memory and Language, after the journal implemented an open data and code policy. Hardwicke and colleagues (2021) reported that they were only able to reproduce the results of 15 of 25 papers in Psychological Science that received open data badges.…”

Section: Introductionmentioning

confidence: 99%

Promoting and Enabling Reproducible Data Science Through a Reproducibility Challenge

Liu

Carlson

Pasek

et al. 2022

Harvard Data Science Review

View full text Add to dashboard Cite

The reproducibility of research results is the basic requirement for the reliability of scientific discovery, yet it is hard to achieve. Whereas funding agencies, scientific journals, and professional societies are developing guidelines, requirements, and incentives, and researchers are developing tools and processes, the role of a university in promoting and enabling reproducible research has been unclear. In this report, we describe the Reproducibility Challenge that we organized at the University of Michigan to promote reproducible research in data science and Artificial Intelligence (AI). We observed that most researchers focused on nuts-and-bolts reproducibility issues relevant to their own research. Many teams were building their own reproducibility protocols and software for lack of any available options off the shelf. If we could help them with this, they would have preferred to adopt rather than build anew. We argue that universities-their data science centers and research support units-have a critical role to play in promoting "actionable reproducibility" (Goeva et al., 2020) through creating and validating tools and processes, and subsequently enabling and encouraging their adoption.

show abstract

Share the code, not just the data: A case study of the reproducibility of articles published in the Journal of Memory and Language under the open data policy

Cited by 22 publications

References 172 publications

Reproducibility and Scientific Integrity of Big Data Research in Urban Public Health and Digital Epidemiology: A Call to Action

Reproducibility and Scientific Integrity of Big Data Research in Urban Public Health and Digital Epidemiology: A Call to Action

How do you Behave as a Psychometrician? Research Conduct in the Context of Psychometric Research

Promoting and Enabling Reproducible Data Science Through a Reproducibility Challenge

Contact Info

Product

Resources

About