What’s in a Badge? A Computational Reproducibility Investigation of the Open Data Badge Policy in One Issue of <i>Psychological Science</i>

Crüwell, Sophia; Apthorp, Deborah; Baker, Bradley J.; Colling, Lincoln; Elson, Malte; Geiger, Sandra J.; Lobentanzer, Sebastian; Monéger, Jean; Patterson, Alex; Schwarzkopf, D. Samuel; Zaneva, Mirela; Brown, Nicholas J. L.

doi:10.1177/09567976221140828

Cited by 23 publications

(17 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The findings of our study are consistent with previous research indicating that open data badges or mandatory open data policies do not necessarily guarantee actual data availability or independent computational reproducibility (Crüwell et al, 2023; Hardwicke et al, 2018, 2021). In the current study, we did not contact any of the original authors if the data were unavailable or were not accompanied by sufficient information to be reusable, but we assume the number of reproducibility checks and MI tests we could perform would have been higher if we had done so.…”

Section: Discussionsupporting

confidence: 91%

The dire disregard of measurement invariance testing in psychological science.

Maassen,

D'Urso,

van Assen

et al. 2023

Psychological Methods

View full text Add to dashboard Cite

Self-report scales are widely used in psychology to compare means in latent constructs across groups, experimental conditions, or time points. However, for these comparisons to be meaningful and unbiased, the scales must demonstrate measurement invariance (MI) across compared time points or (experimental) groups. MI testing determines whether the latent constructs are measured equivalently across groups or time, which is essential for meaningful comparisons. We conducted a systematic review of 426 psychology articles with openly available data, to (a) examine common practices in conducting and reporting of MI testing, (b) assess whether we could reproduce the reported MI results, and (c) conduct MI tests for the comparisons that enabled sufficiently powerful MI testing. We identified 96 articles that contained a total of 929 comparisons. Results showed that only 4% of the 929 comparisons underwent MI testing, and the tests were generally poorly reported. None of the reported MI tests were reproducible, and only 26% of the 174 newly performed MI tests reached sufficient (scalar) invariance, with MI failing completely in 58% of tests. Exploratory analyses suggested that in nearly half of the comparisons where configural invariance was rejected, the number of factors differed between groups. These results indicate that MI tests are rarely conducted and poorly reported in psychological studies. We observed frequent violations of MI, suggesting that reported differences between (experimental) groups may not be solely attributed to group differences in the latent constructs. We offer recommendations aimed at improving reporting and computational reproducibility practices in psychology.

show abstract

Section: Discussionsupporting

confidence: 91%

The dire disregard of measurement invariance testing in psychological science.

Maassen,

D'Urso,

van Assen

et al. 2023

Psychological Methods

View full text Add to dashboard Cite

show abstract

“…(e.g., Foster & Deardorff, 2017; McKiernan et al, 2016; Open Science Collaboration, 2015; Pineau et al, 2021) in order to better support reproducibility and replicability. These principles also extend to computational reproducibility , that focuses on aspects such as the documentation of code, the software environment, and version control that may also impact the replicability and reproducibility of findings (e.g., Crüwell et al, 2023; Stodden et al, 2018). 10 In the context of personality research, encouraging researchers to adopt Open Science practices, and to prioritize the creation, testing, and reporting of reproducible solutions could therefore revolutionize the way personality is assessed and understood, irrespective of the discipline it is reported in.…”

Section: Discussionmentioning

confidence: 99%

“…Open Science practices are also increasingly being required as part of journals' submission policies, for example, seeCrüwell et al (2023) andBrysbaert et al (2021). Thus, researchers would be well-advised not only to adopt these practices to improve reproducibility and replicability, but also to ensure that their work aligns with changes in journal/conference policies which will likely impact all disciplines and outlets eventually.PREDICTING PERSONALITY FROM DIGITAL DATA…”

mentioning

confidence: 99%

Digital data and personality: A systematic review and meta-analysis of human perception and computer prediction.

Hinds,

Joinson

2024

Psychological Bulletin

View full text Add to dashboard Cite

In recent years, our increasing use of technology has resulted in the production of vast amounts of data. Consequently, many researchers have analyzed digital data in attempt to understand its relationship with individuals’ personalities. Such endeavors have inspired efforts from divergent fields, resulting in widely dispersed findings that are seldom synthesized. In this two-part study, we draw from two distinct areas of personality prediction across psychology and computer science to explore the convergent validity of self-reports with human perception and machine learning algorithms, the identifiability of the Big Five traits, and the predictability of different types of data. In Study 1, five meta-analyses of human perception studies integrating findings from 24,124 individuals rated across 30 independent samples demonstrated moderate convergent validity across all traits (ranging from ρ = 0.38 for Neuroticism, to ρ = 0.57 for Openness). In Study 2, a multilevel meta-analysis of computer prediction studies reporting 534 effect sizes across 42 studies also demonstrated moderate convergent validity (ρ = 0.30). Multivariate analyses of the significant moderators highlighted that X, Facebook, Sina Weibo, videos, and smartphones had a negative impact on the variance identified. Finally, in synthesizing the extant literature, we discuss the measures used to assess personality and the analytical approaches adopted. We identify the strengths and limitations across each field and explain how interdisciplinary methodologies could advance the testing and development of psychological theory.

show abstract

“…Crüwell et al [ 15 ] evaluate the computational reproducibility of 14 articles published in Psyhocological Science . Among these articles, the paper by Hilgard et al [ 16 ] has been rated as having “package dependency issues”.…”

Section: Case Studiesmentioning

confidence: 99%

rang: Reconstructing reproducible R computational environments

Chan¹,

David²

2023

PLoS ONE

View full text Add to dashboard Cite

A complete declarative description of the computational environment is usually missing when researchers share their materials. Without such description, software obsolescence and missing system components can jeopardize computational reproducibility in the future, even when data and computer code are available. The R package rang is a complete solution for generating the declarative description for other researchers to automatically reconstruct the computational environment at a specific time point. The reconstruction process, based on Docker, has been tested for R code as old as 2001. The declarative description generated by rang satisfies the definition of a reproducible research compendium and can be shared as such. In this contribution, we show how rang can be used to make otherwise unexecutable code, spanning fields such as computational social science and bioinformatics, executable again. We also provide instructions on how to use rang to construct reproducible and shareable research compendia of current research. The package is currently available from CRAN (https://cran.r-project.org/web/packages/rang/index.html) and GitHub (https://github.com/chainsawriot/rang).

show abstract

What’s in a Badge? A Computational Reproducibility Investigation of the Open Data Badge Policy in One Issue of Psychological Science

Cited by 23 publications

References 44 publications

The dire disregard of measurement invariance testing in psychological science.

The dire disregard of measurement invariance testing in psychological science.

Digital data and personality: A systematic review and meta-analysis of human perception and computer prediction.

rang: Reconstructing reproducible R computational environments

Contact Info

Product

Resources

About