Deciding what to replicate: A decision model for replication study selection under resource and knowledge constraints.

Isager, Peder Mortvedt; Aert, Robbie C. M. van; Bahník, Štěpán; Brandt, Mark J.; DeSoto, K. Andrew; Giner‐Sorolla, Roger; Krueger, Joachim I.; Perugini, Marco; Ropovik, Ivan; Veer, Anna E. van ‘t; Vranka, Marek; Lakens, Daniël

doi:10.31222/osf.io/2gurz

Cited by 18 publications

(42 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In fact, a team science approach could be targeted at different stages of the research process. For example, researchers in a specific domain could crowdsource the most relevant original hypotheses to test or existing findings to replicate, although more princpled ways to arrive at the latter have also been proposed (e.g., Field, Hoekstra, Bringmann, & van Ravenzwaaij 2019 ; Isager et al, 2020 ). I already discussed the project by Landy and colleagues ( 2020 ) who crowdsourced study designs to test a set of hypotheses.…”

Section: Discussionmentioning

confidence: 99%

Two Challenges to “Embodied Cognition” Research And How to Overcome Them

Zwaan

2021

Journal of Cognition

View full text Add to dashboard Cite

From the time the notion "embodied cognition" has entered the field, researchers have been concerned about its meaning. Does the term refer to a coherent theoretical framework? Despite these concerns, use of the term "embodied cognition" has increased over the years to plateau in recent years. I will argue that the best way forward is not to search for evidence for or against some vague label but rather to systematically, in large-scale projects, address a series of questions that focus on welldefined cognitive tasks. Such projects ought involve preregistration, replication, and open materials, code, and data. For this enterprise to take off, it is important that incentives in the field be aligned with the goal to increase the reliability and validity of our research. There is reason to be optimistic that such an alignment will occur in the near future.

show abstract

Section: Discussionmentioning

confidence: 99%

Two Challenges to “Embodied Cognition” Research And How to Overcome Them

Zwaan

2021

Journal of Cognition

View full text Add to dashboard Cite

show abstract

“…Although lower error rates would establish claims more convincingly, this would also require more resources. One might speculate that in research areas where not every claim is important enough to warrant investing the resources required to establish claims with low error rates (Isager et al, 2020), an alpha level of 5% has a pragmatic function in facilitating conjectures and refutations in fields that otherwise lack a coordinated approach to knowledge generation, but are faced with limited resources.…”

Section: The Convention Of An Alpha Level Of 005mentioning

confidence: 99%

The Epistemic and Pragmatic Function of Dichotomous Claims Based on Statistical Hypothesis Tests

Tunç¹,

Mn²,

Lakens³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Researchers commonly make dichotomous claims based on continuous test statistics. Many have branded the practice as misuse of statistics, and criticize scientists for suffering from “dichotomania”. However, the role dichotomous claims play in science is not primarily a statistical one, but an epistemological and pragmatic one. The epistemological function of dichotomous claims consists in transforming data into factual statements that can falsify a universal statement. This transformation requires pre-specified methodological decision procedures such as statistical hypothesis testing (e.g., Neyman-Pearson tests). From the perspective of methodological falsificationism these decision procedures are necessary, as probabilistic statements (e.g. continuous test statistics) cannot function as falsifiers of substantive hypotheses. However, they are not sufficient since for dichotomous claims to have any implication regarding theoretical claims about phenomena, there should be a valid derivation chain linking theoretical, experimental and data models. The pragmatic function of dichotomous claims is facilitating scrutiny and criticism among peers by generating contestable statements, a process referred to by Popper as 'conjectures and refutations', through which we can determine which theories withstand scrutiny the best. Abandoning dichotomous claims to combat the misuse of statistics would not improve scientific inferences but will sacrifice these crucial epistemic and pragmatic functions.

show abstract

“…Pittelkow et al [38] follow a similar procedure, applied to studies from clinical psychology. Isager et al [39] outline a model for deciding on the utility of replicating a study, given known costs, by calculating the value of a claim and the uncertainty about that claim prior to replication. The key variables in this model -costs, value, and uncertainty -remain undefined, with the expectation that each can be specified outside the model (as relevant to a given knowledge domain).…”

Section: Replicats As a Model For Allocating Replication Effortmentioning

confidence: 99%

Predicting reliability through structured expert elicitation with repliCATS (Collaborative Assessments for Trustworthy Science)

Fraser¹,

Bush²,

Wintle³

et al. 2021

Preprint

View full text Add to dashboard Cite

Replication is a hallmark of scientific research. As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce a new technique to evaluating replicability, the repliCATS (Collaborative Assessments for Trustworthy Science) process, a structured expert elicitation approach based on the IDEA protocol. The repliCATS process is delivered through an underpinning online platform and applied to the evaluation of research claims in social and behavioural sciences. This process can be deployed for both rapid assessment of small numbers of claims, and assessment of high volumes of claims over an extended period. Pilot data suggests that the accuracy of the repliCATS process meets or exceeds that of other techniques used to predict replicability. An important advantage of the repliCATS process is that it collects qualitative data that has the potential to assist with problems like understanding the limits of generalizability of scientific claims. The repliCATS process has potential applications in alternative peer review and in the allocation of effort for replication studies.

show abstract

Deciding what to replicate: A decision model for replication study selection under resource and knowledge constraints.

Cited by 18 publications

References 55 publications

Two Challenges to “Embodied Cognition” Research And How to Overcome Them

Two Challenges to “Embodied Cognition” Research And How to Overcome Them

The Epistemic and Pragmatic Function of Dichotomous Claims Based on Statistical Hypothesis Tests

Predicting reliability through structured expert elicitation with repliCATS (Collaborative Assessments for Trustworthy Science)

Contact Info

Product

Resources

About