Is there a reproducibility crisis around here? Maybe not, but we still need to change.

Holcombe, Alex O.; Ludowici, Charles; Haroz, Steve

doi:10.1167/19.10.87a

Cited by 3 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When looking at these practices in isolation, this research suggests a relatively low uptake, as well as variability across psychology subdisciplines. For example, Holcombe et al (2019) note that for vision sciences, engagement in practices such as open data and open code is low. Towse et al (2021) show that while data sharing is low across the board in psychology (at around 4%), data sharing in social psychology journals is higher than that observed for cognitive science or applied psychology journals.…”

Section: Responsible Research Practices In Psychologymentioning

confidence: 99%

See 1 more Smart Citation

A registered report survey of open research practices in psychology departments in the UK and Ireland

Silverstein,

Pennington,

Branney

et al. 2024

British J of Psychology

View full text Add to dashboard Cite

Open research practices seek to enhance the transparency and reproducibility of research. While there is evidence of increased uptake in these practices, such as study preregistration and open data, facilitated by new infrastructure and policies, little research has assessed general uptake of such practices across psychology university researchers. The current study estimates psychologists' level of engagement in open research practices across universities in the United Kingdom and Ireland, while also assessing possible explanatory factors that may impact their engagement. Data were collected from 602 psychology researchers in the United Kingdom and Ireland on the extent to which they have implemented various practices (e.g., use of preprints, preregistration, open data, open materials). Here we present the summarized descriptive results, as well as considering differences between various categories of researcher (e.g., career stage, subdiscipline, methodology), and examining the relationship between researcher's practices and their self‐reported capability, opportunity, and motivation (COM‐B) to engage in open research practices. Results show that while there is considerable variability in engagement of open research practices, differences across career stage and subdiscipline of psychology are small by comparison. We observed consistent differences according to respondent's research methodology and based on the presence of institutional support for open research. COM‐B dimensions were collectively significant predictors of engagement in open research, with automatic motivation emerging as a consistently strong predictor. We discuss these findings, outline some of the challenges experienced in this study, and offer suggestions and recommendations for future research. Estimating the prevalence of responsible research practices is important to assess sustained behaviour change in research reform, tailor educational training initiatives, and to understand potential factors that might impact engagement.

show abstract

Section: Responsible Research Practices In Psychologymentioning

confidence: 99%

“…For example, Holcombe et al. (2019) note that for vision sciences, engagement in practices such as open data and open code is low. Towse et al.…”

Section: Problems For Science and Psychologymentioning

confidence: 99%

A registered report survey of open research practices in psychology departments in the UK and Ireland

Silverstein,

Pennington,

Branney

et al. 2024

British J of Psychology

View full text Add to dashboard Cite

show abstract

Section: Responsible Research Practices In Psychologymentioning

confidence: 99%

A Registered Report Survey of Open Research Practices in Psychology Departments in the UK and Ireland

Silverstein¹,

Pennington²,

Branney³

et al. 2023

Preprint

View full text Add to dashboard Cite

Open research practices seek to enhance the transparency and reproducibility of research. Whilst there is evidence of increased uptake in these practices, such as study preregistration and open data, facilitated by new infrastructure and policies, little research has assessed general uptake of such practices across university researchers. The current study estimates psychologists’ level of engagement in open research practices across universities in the United Kingdom and Ireland, while also assessing possible explanatory factors that may impact their engagement. Data were collected from [insert final number of participants] psychology researchers in the UK and Ireland on the extent to which individual researchers have implemented various practices (e.g., use of preprints, preregistration, open data, open materials). Here we present the summarised descriptive results, as well as considering differences between various categories of researcher (e.g., career stage, subdiscipline, methodology), and examining the link between researcher’s practices and their self-reported capability, opportunity, and motivation (COM-B) to engage in open research practices. Results show [e.g., that there is wide/little variability across the discipline as well as differences between established researchers and postgraduate research students/postdoctoral researchers, who are more/less likely to be engaging in responsible research practices]. Estimating the prevalence of responsible research practices is important to assess sustained behaviour change in research reform, tailor educational training initiatives, and to understand potential factors that might impact engagement.

show abstract

“…In some respects, vision scientists believe their work to be so robust as to be immune to the issues of reproducibility that have plagued other fields within psychology (e.g. Holcombe, Ludowici, & Haroz, 2019 ). Our effects are generally quite replicable ( Zwaan, Pecher, Paolacci, Bouwmeester, Verkoeijen, Dijkstra, & Zeelenberg, 2018 ), and we tend to think that our measures are telling us something meaningful about what is happening perceptually, cognitively, and/or neurally.…”

Section: Introductionmentioning

confidence: 99%

Test-retest reliability for common tasks in vision science

et al. 2022

View full text Add to dashboard Cite

Research in perception and attention has typically sought to evaluate cognitive mechanisms according to the average response to a manipulation. Recently, there has been a shift toward appreciating the value of individual differences and the insight gained by exploring the impacts of between-participant variation on human cognition. However, a recent study suggests that many robust, well-established cognitive control tasks suffer from surprisingly low levels of test-retest reliability (Hedge, Powell, & Sumner, 2018b). We tested a large sample of undergraduate students ( n = 160) in two sessions (separated by 1–3 weeks) on four commonly used tasks in vision science. We implemented measures that spanned a range of perceptual and attentional processes, including motion coherence (MoCo), useful field of view (UFOV), multiple-object tracking (MOT), and visual working memory (VWM). Intraclass correlations ranged from good to poor, suggesting that some task measures are more suitable for assessing individual differences than others. VWM capacity (intraclass correlation coefficient [ICC] = 0.77), MoCo threshold (ICC = 0.60), UFOV middle accuracy (ICC = 0.60), and UFOV outer accuracy (ICC = 0.74) showed good-to-excellent reliability. Other measures, namely the maximum number of items tracked in MOT (ICC = 0.41) and UFOV number accuracy (ICC = 0.48), showed moderate reliability; the MOT threshold (ICC = 0.36) and UFOV inner accuracy (ICC = 0.30) showed poor reliability. In this paper, we present these results alongside a summary of reliabilities estimated previously for other vision science tasks. We then offer useful recommendations for evaluating test-retest reliability when considering a task for use in evaluating individual differences.

show abstract

Is there a reproducibility crisis around here? Maybe not, but we still need to change.

Cited by 3 publications

References 0 publications

A registered report survey of open research practices in psychology departments in the UK and Ireland

A registered report survey of open research practices in psychology departments in the UK and Ireland

A Registered Report Survey of Open Research Practices in Psychology Departments in the UK and Ireland

Test-retest reliability for common tasks in vision science

Contact Info

Product

Resources

About