Assessing the replication landscape in experimental linguistics

Kobrock, Kristina; Roettger, Timo B.

doi:10.31234/osf.io/fzngs

Cited by 3 publications

(6 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To this end, Brandt et al (2014) have provided a preregistration template specifically designed for replication studies. Open multisite replication projects are another promising way forward for replication (Kobrock & Roettger, 2022), whereby groups of researchers agree on what to replicate and how to do so, preregister the protocol, and collect data at multiple sites. Transparency and open practices are essential and inherent to such endeavors, and free infrastructure and tools are available to facilitate large-teams working.…”

Section: In Language Learning) But How Might Open Research Help?mentioning

confidence: 99%

(Why) Are Open Research Practices the Future for the Study of Language Learning?

Marsden

Morgan‐Short²

2023

Language Learning

View full text Add to dashboard Cite

Open research practices are relevant to all stages of research, from conceptualization through dissemination. Here, we discuss key facets of open research, highlighting its rationales, infrastructures, behaviors, and challenges. Part I conceptualizes open research and its rationales. Part II identifies challenges such as the speed and cost of open research, the usability of open data and materials, the difficulties of conducting replication research, and the economics and sustainability of open access and open research generally. In discussing these challenges, we have sought to provide examples of good practice, describe and evaluate emerging innovations, and envision change. Part III considers ongoing coevolutions of culture, infrastructure, and behaviors and acknowledges the limitations of our review and of open research practices. We argue that open research is indeed a large part of our future, and most—if not all—challenges are surmountable, but doing so requires significant changes for many aspects of the research process.

show abstract

Section: In Language Learning) But How Might Open Research Help?mentioning

confidence: 99%

(Why) Are Open Research Practices the Future for the Study of Language Learning?

Marsden

Morgan‐Short²

2023

Language Learning

View full text Add to dashboard Cite

show abstract

“…To this end, Brandt et al (2014) provide a preregistration template specifically designed for replication studies. Open multi-site replication projects are another promising way forward for replication (Kobrock & Roettger, 2022), whereby groups of researchers agree on what to replicate and how to do so, preregister the protocol, and collect data at multiple sites. Transparency and open practices are essential and inherent to such endeavors, and free infrastructure and tools are even available to facilitate large-teams working.…”

Section: In Language Learning) But How Might Open Research Help?mentioning

confidence: 99%

“…As the field adopts Open Research practices to enhance replication efforts, we should continue to dialogue about what to replicate and why. Several perspectives on these issues have been offered both within and outside the language sciences (e.g., Field et al, 2019;Isager et al, 2021;Kobrock & Roettger, 2022;. Central among these perspectives is the need for replication to be driven by the contribution and impact of the initial and replication studies, rather than by the availability of open materials per se.…”

Section: In Language Learning) But How Might Open Research Help?mentioning

confidence: 99%

(Why) Are Open Research Practices the Future for the Study of Language Learning?

Marsden¹,

Morgan‐Short²

2023

Preprint

View full text Add to dashboard Cite

Open Research practices are relevant to all stages of research, from conceptualisation through to dissemination. Here, we discuss key facets of Open Research, highlighting its rationales, infrastructures, behaviours, and challenges. Part I conceptualizes Open Research and its rationales. Part II identifies challenges, such as the speed and cost of Open Research; the usability of open data and materials; the difficulties of conducting replication research; and the economics and sustainability of open access and Open Research generally. In discussing these challenges, we seek to provide examples of good practice, describe and evaluate emerging innovations, and envision change. Part III considers ongoing co-evolutions of culture, infrastructure, and behaviours, and acknowledges the limitations of our review and of Open Research practices. We argue that Open Research is indeed a large part of our future, and most—if not all—challenges are surmountable, but doing so requires significant changes for many aspects of the research process.

show abstract

“…However, there is little evidence on the adoption of transparency and reproducibility practices in this field. Recent meta-scientific assessments have investigated important aspects of research practices in language research, including the prevalence of direct replications (50,51) and analytical flexibility (52). Moreover, a recent assessment which targeted a specific psycholinguistic journal suggests a general lack of data sharing and reproducibility (27).…”

Section: Introductionmentioning

confidence: 99%

“…Following Kobrock and Roettger(50), we first searched for the search string "replicat" and if there was a hit, we examined the title and the abstract of the paper, the text before and after occurrences of the search term "replicat", the paragraph before the Methods section as well as the first paragraph of the Discussion section. If the authors explicitly claimed that (one of) their research aim(s) was to replicate the result or methods of an initial study, this article was treated as a replication.…”

mentioning

confidence: 99%

Reproducible research practices and transparency across linguistics

Bochynska¹,

Keeble²,

Halfacre³

et al. 2023

Preprint

View full text Add to dashboard Cite

Scientific studies of language span across many disciplines and provide evidence for social, cultural, cognitive, technological, and biomedical studies of human nature and behavior. By becoming increasingly empirical and quantitative, linguistics has been facing challenges and limitations of the scientific practices that pose barriers to reproducibility and replicability. One of the proposed solutions to the widely acknowledged reproducibility and replicability crisis has been the implementation of transparency practices, e.g. open access publishing, preregistrations, sharing study materials, data, and analyses, performing study replications and declaring conflicts of interest. Here, we have assessed the prevalence of these practices in randomly sampled 600 journal articles from linguistics across two time points. In line with similar studies in other disciplines, we found a moderate amount of articles published open access, but overall low rates of sharing materials, data, and protocols, no preregistrations, very few replications and low rates of conflict of interest reports. These low rates have not increased noticeably between 2008/2009 and 2018/2019, pointing to remaining barriers and slow adoption of open and reproducible research practices in linguistics. As linguistics has not yet firmly established transparency and reproducibility as guiding principles in research, we provide recommendations and solutions for facilitating the adoption of these practices.

show abstract

Assessing the replication landscape in experimental linguistics

Cited by 3 publications

References 37 publications

(Why) Are Open Research Practices the Future for the Study of Language Learning?

(Why) Are Open Research Practices the Future for the Study of Language Learning?

(Why) Are Open Research Practices the Future for the Study of Language Learning?

Reproducible research practices and transparency across linguistics

Contact Info

Product

Resources

About