Replication in Experimental Communication Research: An Analysis

Kelly, Clifford W.; Chase, Lawrence J.; Tucker, Raymond K.

doi:10.1111/j.1468-2958.1979.tb00646.x

Cited by 50 publications

(54 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…On the basis of Makel et al's (2012) article, replications (at least published ones) in psychological science are extremely uncommon-in the same range as previously documented for economics, marketing, and communication (Evanschitzky, Baumgarth, Hubbard, & Armstrong, 2006;Hubbard & Armstrong, 1994;Kelly, Vhase, & Tucker, 1979). Makel et al (2012) screened all the papers published by the 100 most-cited psychology journals since 1900 and they estimated that only 1.07% of these papers represented replication efforts.…”

Section: Possible Pairs Of Discovery and Replication Results And Theimentioning

confidence: 91%

Why Science Is Not Necessarily Self-Correcting

Ioannidis

2012

Perspect Psychol Sci

454

383

View full text Add to dashboard Cite

The ability to self-correct is considered a hallmark of science. However, self-correction does not always happen to scientific evidence by default. The trajectory of scientific credibility can fluctuate over time, both for defined scientific fields and for science at-large. History suggests that major catastrophes in scientific credibility are unfortunately possible and the argument that "it is obvious that progress is made" is weak. Careful evaluation of the current status of credibility of various scientific fields is important in order to understand any credibility deficits and how one could obtain and establish more trustworthy results. Efficient and unbiased replication mechanisms are essential for maintaining high levels of scientific credibility. Depending on the types of results obtained in the discovery and replication phases, there are different paradigms of research: optimal, self-correcting, false nonreplication, and perpetuated fallacy. In the absence of replication efforts, one is left with unconfirmed (genuine) discoveries and unchallenged fallacies. In several fields of investigation, including many areas of psychological science, perpetuated and unchallenged fallacies may comprise the majority of the circulating evidence. I catalogue a number of impediments to self-correction that have been empirically studied in psychological science. Finally, I discuss some proposed solutions to promote sound replication practices enhancing the credibility of scientific results as well as some potential disadvantages of each of them. Any deviation from the principle that seeking the truth has priority over any other goals may be seriously damaging to the self-correcting functions of science.

show abstract

Section: Possible Pairs Of Discovery and Replication Results And Theimentioning

confidence: 91%

Why Science Is Not Necessarily Self-Correcting

Ioannidis

2012

Perspect Psychol Sci

454

383

View full text Add to dashboard Cite

show abstract

“…In the field of psychology, Makel et al (2012) estimated that among the top 100 journals between 1900 and 2010, the replication study publication rate was 1.07%, though this rate is now likely to be higher given recent multiple, direct replication projects: the Many Labs project (Klein et al, 2014), the Pipeline Project (Schweinsberg et al, 2016), the Registered Reports project (Nosek & Lakens, 2014), and the Reproducibility Project (Open Science Collaboration, 2015). In business, marketing, and communication journals, replication rates have ranged from 1 to 3% (Evanschitzky, Baumgarth, Hubbard, & Armstrong, 2007;Hubbard & Armstrong, 1994;Kelly, Chase, & Tucker, 1979). In the field of L2 research, the rate of replication studies is perceived as being low, but without systematic data on this, concerns to date have necessarily been speculative.…”

Section: The Quantity Of Replication Researchmentioning

confidence: 99%

Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field

et al. 2018

View full text Add to dashboard Cite

Despite its critical role for the development of the field, little is known about replication in second language (L2) research. To better understand replication practice, we first provide a narrative review of challenges related to replication, drawing on recent developments in psychology. This discussion frames and motivates a systematic review, building on syntheses of replication in psychology, education, and L2 research. We coded 67 self-labeled L2 replication studies found across 26 journals for 136 characteristics. We estimated a mean rate of 1 published replication study for every 400 articles, with a mean of 6.64 years between initial and replication studies and a mean of 117 citations of the initial study before a replication was published. Replication studies had an annual mean of 7.3 citations, much higher than averages in linguistics and education. Overlap in authorship between initial and replication studies and the availability of the initial materials both increased the likelihood of a replication supporting the initial findings. Our sample contained no direct (exact) replication attempts, and changes made to initial studies were numerous and wide ranging, which likely obscured, if not undermined, the interpretability of replication studies. To improve the amount and quality of L2 replication research, we propose 16 recommendations relating to rationale, nomenclature, design, infrastructure, and incentivization for collaboration and publication.

show abstract

“…A dimension that might be referred to as operationalization is suggested by Adams et al [1] in their operational and instrumental replications, Barker and Gurman [4] in their Type III replication, Finifter [20] in his virtual replication, Kelly et al [36] in their operational replication, or Schmidt [61] in his direct replication.…”

Section: Elements Of An Experimental Configurationmentioning

confidence: 99%

“…when the analysis of the sample data gathered (experiment) shows a significant difference between the treatments that it compares, but no such difference exists in the population (reality). Bahr et al [3] Kelly et al [36] Finifter 1975 Finifter [21] Lykken [43] Sidman [64] La Sorte [40] Blomquist [9] X …”

mentioning

confidence: 99%

Understanding replication of experiments in software engineering: A classification

Gómez

Juristo

Vegas

2014

Information and Software Technology

109

117

View full text Add to dashboard Cite

Context: Replication plays an important role in experimental disciplines. There are still many uncertain-ties about how to proceed with replications of SE experiments. Should replicators reuse the baseline experiment materials? How much liaison should there be among the original and replicating experiment-ers, if any? What elements of the experimental configuration can be changed for the experiment to be considered a replication rather than a new experiment? Objective: To improve our understanding of SE experiment replication, in this work we propose a classi-fication which is intend to provide experimenters with guidance about what types of replication they can perform. Method: The research approach followed is structured according to the following activities: (1) a litera-ture review of experiment replication in SE and in other disciplines, (2) identification of typical elements that compose an experimental configuration, (3) identification of different replications purposes and (4) development of a classification of experiment replications for SE. Results: We propose a classification of replications which provides experimenters in SE with guidance about what changes can they make in a replication and, based on these, what verification purposes such a replication can serve. The proposed classification helped to accommodate opposing views within a broader framework, it is capable of accounting for less similar replications to more similar ones regarding the baseline experiment. Conclusion: The aim of replication is to verify results, but different types of replication serve special ver-ification purposes and afford different degrees of change. Each replication type helps to discover partic-ular experimental conditions that might influence the results. The proposed classification can be used to identify changes in a replication and, based on these, understand the level of verification.

show abstract

Replication in Experimental Communication Research: An Analysis

Cited by 50 publications

References 8 publications

Why Science Is Not Necessarily Self-Correcting

Why Science Is Not Necessarily Self-Correcting

Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field

Understanding replication of experiments in software engineering: A classification

Contact Info

Product

Resources

About