The Importance of Standards for Sharing of Computational Models and Data

Poldrack, Russell A.; Feingold, Franklin; Frank, Michael J.; Gleeson, Padraig; Hollander, Gilles de; Huys, Quentin J. M.; Love, Bradley C.; Markiewicz, Christopher J.; Moran, Rosalyn J.; Ritter, Petra; Rogers, Timothy T.; Turner, B. E.; Yarkoni, Tal; Zhan, Ming; Cohen, Jonathan D.

doi:10.1007/s42113-019-00062-x

Cited by 12 publications

(6 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, while it may superficially appear that we are at odds with the emphasis on the bottom few steps in our path model (hypothesis testing and data analysis, recall Figure 2) by those who are investigating replicability, we are comfortable with this emphasis. We believe the proposals set out by some to automate or streamline the last few steps are part of the solution (e.g., Lakens & DeBruine, 2020;Poldrack et al, 2019). Such a division of labor, might help maximise the quality of theories and showcase the contrast -which Meehl (1967) and others have drawn attention to -between substantive theories and the hypotheses they generate.…”

Section: A Way Forwardmentioning

confidence: 99%

How Computational Modeling Can Force Theory Building in Psychological Science

Guest

Martin

2021

Perspect Psychol Sci

242

237

View full text Add to dashboard Cite

Psychology endeavors to develop theories of human capacities and behaviors on the basis of a variety of methodologies and dependent measures. We argue that one of the most divisive factors in psychological science is whether researchers choose to use computational modeling of theories (over and above data) during the scientific-inference process. Modeling is undervalued yet holds promise for advancing psychological science. The inherent demands of computational modeling guide us toward better science by forcing us to conceptually analyze, specify, and formalize intuitions that otherwise remain unexamined—what we dub open theory. Constraining our inference process through modeling enables us to build explanatory and predictive theories. Here, we present scientific inference in psychology as a path function in which each step shapes the next. Computational modeling can constrain these steps, thus advancing scientific inference over and above the stewardship of experimental practice (e.g., preregistration). If psychology continues to eschew computational modeling, we predict more replicability crises and persistent failure at coherent theory building. This is because without formal modeling we lack open and transparent theorizing. We also explain how to formalize, specify, and implement a computational model, emphasizing that the advantages of modeling can be achieved by anyone with benefit to all.

show abstract

Section: A Way Forwardmentioning

confidence: 99%

How Computational Modeling Can Force Theory Building in Psychological Science

Guest

Martin

2021

Perspect Psychol Sci

242

237

View full text Add to dashboard Cite

show abstract

“…In neuroscience specifically, platforms such as OpenNeuro (https://openneuro.org/) have been developed to openly share neuroimaging datasets, and the 'Brain Imaging Data Structure' (BIDS; https://bids.neuroimaging.io/; has been developed to homogenize the structure of datasets, which fosters time-efficient use of datasets and development of code tailored to this format (see, e.g., fMRIPrep, Esteban et al, 2019;and MRIQC, Esteban et al, 2017), allowing the same code to be easily applied to multiple datasets. Note that similar homogeneous standards for the structure of computational models have also been proposed (Poldrack et al, 2019). Platforms such as the "NeuroImaging Tools & Resources Collaboratory" (NITRC; https://www.nitrc.org/) are used for data as well as software sharing.…”

Section: Improvement Referencementioning

confidence: 99%

A review of ideas and strategies to improve scientific research and its dissemination in life and social sciences

Almgren¹,

Parsons²,

Bossche³

et al. 2020

Preprint

View full text Add to dashboard Cite

In recent years, issues concerning reproducibility, questionable research practices (QRPs), and analytical flexibility have been a contentious topic in the fields of life and social sciences. As such, research has increasingly focused on studying the way we conduct science (which is a field of study in meta-science), and new ideas have been introduced on how we can improve the way we do science. In this literature review, we aim to summarize research and ideas that have been introduced in the past decade. We give an overview of studies concerning reproducibility in contemporary life and social sciences, and outline QRPs which are often used when results do not support the researcher’s hypothesis and/or when results are not statistically significant. We also discuss cognitive biases in research and the topic of analytical flexibility, which warrants straightforward interpretation and generalization of results. We summarize the literature on improving the scientific process and system of life and social sciences in six broad categories, including (1) Focus on aggregate effects across replication attempts, (2) Achieving more solid empirical claims through (pre)registered reports, (3) Transparency, complete reporting, and openness (if warranted) of data, code, all performed analyses, results, and (changes in) research plans, (4) Expanding our statistical toolbox and justifying analytical choices a priori, (5) Changing incentives in science from a focus on quantity to quality and rigor, and (6) Improving the way we decide what is considered scientific. All new implementations will require effort and time, and some implementations will create new problems which will have to be solved. A systematic evaluation of the impact of new ideas and implementations will help us fine-tune the scientific system in place, and ultimately improve the validity of scientific claims.

show abstract

“…However, the proportion of scientific data that is actually openly shared within the neuroscientific community remains low ( Watson, 2022 ). The lack of sharing properly annotated data and tools contributes to the poor reproducibility of research results, known as “the reproducibility crisis,” that hinders the growth of knowledge and innovation on the one hand and leads to inefficient use of resources on the other hand ( Baker, 2016 ; Stodden et al, 2016 ; Poldrack et al, 2019 ; Crook et al, 2020 ; Loss et al, 2021 ; Niso et al, 2022 ).…”

Section: Introductionmentioning

confidence: 99%

Research Data Management and Data Sharing for Reproducible Research—Results of a Community Survey of the German National Research Data Infrastructure Initiative Neuroscience

Klingner

Denker

Grün

et al. 2023

eNeuro

Self Cite

View full text Add to dashboard Cite

Science is changing: the volume and complexity of data is increasing, the number of studies is growing and the goal of achieving reproducible results requires new solutions for scientific data management. In the field of neuroscience, the German National Research Data Infrastructure (NFDI-Neuro) initiative aims to develop sustainable solutions for research data management (RDM). To obtain an understanding of the present RDM situation in the neuroscience community, NFDI-Neuro conducted a comprehensive survey amongst the neuroscience community. Here, we report and analyse the results of the survey. We focused the survey and our analysis on current needs, challenges, and opinions about RDM.The German neuroscience community perceives barriers with respect to RDM and data sharing mainly linked to (1) lack of data and metadata standards, (2) lack of community adopted provenance tracking methods, 3) lack of secure and privacy preserving research infrastructure for sensitive data (4) lack of RDM literacy and (5) lack of resources (time, personnel, money) for proper RDM. However, an overwhelming majority of community members (91%) indicated that they would be willing to share their data with other researchers and are interested to increase their RDM skills. Taking advantage of this willingness and overcoming the existing barriers requires the systematic development of standards, tools, and infrastructure, the provision of training, education, and support, as well as additional resources for RDM to the research community and a constant dialogue with relevant stakeholders including policy makers to leverage of a culture change through adapted incentivisation and regulation.Significance StatementA comprehensive survey amongst the neuroscience community in Germany determined the current needs, challenges, and opinions with respect to standardized research data management (RDM). The Neuroscience community perceives a lack of standards for data and metadata, a lack of provenance tracking and versioning of data, a lack of protected digital research infrastructure for sensitive data and a lack of education and resources for proper RDM. However, an overwhelming majority of community members indicated that they would be willing to share their data with other researchers and are interested to increase their RDM skills. Thus, the survey results suggest that training, the provision of standards, tools, infrastructure, and resources for RDM holds the potential to significantly facilitate reproducible research in neuroscience.

show abstract

The Importance of Standards for Sharing of Computational Models and Data

Cited by 12 publications

References 17 publications

How Computational Modeling Can Force Theory Building in Psychological Science

How Computational Modeling Can Force Theory Building in Psychological Science

A review of ideas and strategies to improve scientific research and its dissemination in life and social sciences

Research Data Management and Data Sharing for Reproducible Research—Results of a Community Survey of the German National Research Data Infrastructure Initiative Neuroscience

Contact Info

Product

Resources

About