Replication is important for educational psychology: Recent developments and key issues

Plucker, Jonathan A.; Makel, Matthew C.

doi:10.1080/00461520.2021.1895796

Cited by 81 publications

(57 citation statements)

References 89 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1 Naturally, without full access to all measures and analyses, replication attempts are often doomed from the outset. Even if researchers have access to the same measures and similar participants, if the original analyses capitalized on the aforementioned flexibility in "researcher degrees of freedom," the odds of a successful replication are vanishingly small (see Plucker & Makel, 2021/this issue for more on different types of replication studies). Additionally, knowing that journals prefer to publish statistically significant results creates a "file drawer" problem (Rosenthal, 1979) in which researchers are 3, but then bolding only those parts that would typically be reported in an old-school approach.…”

Section: Old-school Science In Contextmentioning

confidence: 99%

“…What types of research methodologies are viable for registered reports (Reich, 2021/this issue)? What expectations are reasonable for replication studies across distinct methodological approaches (Plucker & Makel, 2021/ this issue)? Answers to these types of questions could be helpful for sociologists of education, educational historians, educational policy scholars, and so forth.…”

Section: Winning Over Skeptical Hearts and Mindsmentioning

confidence: 99%

See 1 more Smart Citation

From old school to open science: The implications of new research norms for educational psychology and beyond

Gehlbach

Robinson

2021

Educational Psychologist

View full text Add to dashboard Cite

Recently, scholars have noted how several "old-school" practices-a host of long-standing scientific norms-in combination, sometimes compromise the credibility of research. In response, other scholarly fields have developed several "open-science" norms and practices to address these credibility issues. Against this backdrop, this special issue explores the extent to which and how these norms should be adopted and adapted for educational psychology and education more broadly. Our introductory article contextualizes the special issue's goals by overviewing the historical context that led to open science norms (particularly in medicine and psychology); providing a conceptual map to illustrate the interrelationships between various old-school as well as open-science practices; and then describing educational psychologists' opportunity to benefit from and contribute to the translation of these norms to novel research contexts. We conclude by previewing the articles in the special issue. Adopting and adapting open scienceCognitively, we knew that the mere act of preregistering studies was not enough to ensure the level of rigor we sought in our work. But the painful experiential learning gained from these replication attempts proved to be a much

show abstract

Section: Old-school Science In Contextmentioning

confidence: 99%

Section: Winning Over Skeptical Hearts and Mindsmentioning

confidence: 99%

From old school to open science: The implications of new research norms for educational psychology and beyond

Gehlbach

Robinson

2021

Educational Psychologist

View full text Add to dashboard Cite

show abstract

“…1 Naturally, without full access to all measures and analyses, replication attempts are often doomed from the outset. Even if researchers have access to the same measures and similar participants, if the original analyses capitalized on the aforementioned flexibility in "researcher degrees of freedom," the odds of a successful replication are vanishingly small (see Plucker & Makel, 2021/this issue for more on different types of replication studies). Additionally, knowing that journals prefer to publish statistically significant results creates a "file drawer" problem (Rosenthal, 1979) in which researchers are incentivized to stick their failed studies, including failed replications, into a metaphorical file drawer rather than trying to publish them.…”

Section: Old School Science In Contextmentioning

confidence: 99%

“…this issue)? If measures function differently with different participant populations, how can we evaluate whether a replication attempt has succeeded(Plucker & Makel, 2021/this issue)? Many researchers view their measures as intellectual property and charge fees for their use.…”

mentioning

confidence: 99%

From Old School to Open Science: The Implications of New Research Norms for Educational Psychology and Beyond

Gehlbach¹,

Robinson²

2021

Preprint

View full text Add to dashboard Cite

Recently, scholars have noted how several “old school” practices—a host of well-regarded, long-standing scientific norms—in combination, sometimes compromise the credibility of research. In response, other scholarly fields have developed several “open science” norms and practices to address these credibility issues. Against this backdrop, this special issue explores the extent to which and how these norms should be adopted and adapted for educational psychology and education more broadly. Our introductory article contextualizes the special issue’s goals by: overviewing the historical context that led to open science norms (particularly in medicine and psychology); providing a conceptual map to illustrate the interrelationships between various old school as well as open science practices; and then describing educational psychologists’ opportunity to benefit from and contribute to the translation of these norms to novel research contexts. We conclude by previewing the articles in the special issue.

show abstract

“…After close to a century of repeated reminders that replication studies play an important role in establishing robust scientific knowledge, and following a number of high-profile replications published in the last decade (e.g., Open Science Collaboration, 2015; Hagger et al, 2016;Ranehill et al, 2015;Ritchie, Wiseman, & French, 2012;Wagenmakers et al, 2016), researchers from many scientific disciplines are increasingly calling for a renewed focus on replication research (Blaszczynski & Gainsbury, 2019;Button et al, 2013;Heirene, 2021;Murphy, Mesquida, Caldwell, Earp, & Warne, 2021;Plucker & Makel, 2021;Sale & Mellor, 2018;Zwaan, Etz, Lucas, & Donnellan, 2018). Given the exponential growth of the empirical literature (Bornmann & Mutz, 2015;Parolo et al, 2015) and the low rates of replication up until this point (Makel, Plucker, & Hegarty, 2012;Mueller-Langer, Fecher, Harhoff, & Wagner, 2019), researchers interested in conducting replications of original research will often have to choose which of several replication targets to focus limited resources on.…”

Section: Introductionmentioning

confidence: 99%

Replication value as a function of citation impact and sample size

Isager¹,

Veer²,

Lakens³

2021

Preprint

View full text Add to dashboard Cite

Researchers seeking to replicate original research often need to decide which of several relevant candidates to select for replication. Several strategies for study selection have been proposed, utilizing a variety of observed indicators as criteria for selection. However, few strategies clearly specify the goal of study selection and how that goal is related to the indicators that are utilized. We have previously formalized a decision model of replication study selection in which the goal of study selection is to maximize the expected utility gain of the replication e?ort. We further define the concept of replication value as a proxy for expected utility gain (Isager et al., 2020). In this article, we propose a quantitative operationalization of replication value. Wefirst discuss how value and uncertainty - the two concepts used to determine replication value – could be estimated via information about citation count and sample size. Second, we propose an equation for combining these indicators into an overall estimate of replication value, which we denote RVCn. Third, we suggest how RVCn could be implemented as part of a broader study selection procedure. Finally, we provide preliminary data suggesting that studies that were in fact selected for replication tend to have relatively high RVCn estimates. The goal of this article is to explain how RVCn is intended to work and, in doing so, demonstrate the many assumptions that should be explicit in any replication study selection strategy.

show abstract

Replication is important for educational psychology: Recent developments and key issues

Cited by 81 publications

References 89 publications

From old school to open science: The implications of new research norms for educational psychology and beyond

From old school to open science: The implications of new research norms for educational psychology and beyond

From Old School to Open Science: The Implications of New Research Norms for Educational Psychology and Beyond

Replication value as a function of citation impact and sample size

Contact Info

Product

Resources

About