Enhancing Transparency of the Research Process to Increase Accuracy of Findings: A Guide for Relationship Researchers

Campbell, Lorne; Loving, Timothy J.; LeBel, Etienne P.

doi:10.2139/ssrn.2435185

Cited by 14 publications

(13 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Lily could not have preregistered the entire design and analysis plan for all future papers at project onset. Moreover, the longitudinal design amplifies the challenges of preexisting data and changes to protocols after preregistration (52,53).…”

Section: Preregistration In Practicementioning

confidence: 99%

The preregistration revolution

Nosek

Ebersole

DeHaven

et al. 2018

Proc. Natl. Acad. Sci. U.S.A.

1,437

1,267

View full text Add to dashboard Cite

Progress in science relies in part on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated conceptually but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings. However, ordinary biases in human reasoning, such as hindsight bias, make it hard to avoid this mistake. An effective solution is to define the research questions and analysis plan before observing the research outcomes-a process called preregistration. Preregistration distinguishes analyses and outcomes that result from predictions from those that result from postdictions. A variety of practical strategies are available to make the best possible use of preregistration in circumstances that fall short of the ideal application, such as when the data are preexisting. Services are now available for preregistration across all disciplines, facilitating a rapid increase in the practice. Widespread adoption of preregistration will increase distinctiveness between hypothesis generation and hypothesis testing and will improve the credibility of research findings.methodology | open science | confirmatory analysis | exploratory analysis | preregistration P rogress in science is marked by reducing uncertainty about nature. Scientists generate models that may explain prior observations and predict future observations. Those models are approximations and simplifications of reality. Models are iteratively improved and replaced by reducing the amount of prediction error. As prediction error decreases, certainty about what will occur in the future increases. This view of research progress is captured by George Box's aphorism: "All models are wrong but some are useful" (1, 2).Scientists improve models by generating hypotheses based on existing observations and testing those hypotheses by obtaining new observations. These distinct modes of research are discussed by philosophers and methodologists as hypothesis-generating versus hypothesis-testing, the context of discovery versus the context of justification, data-independent versus data-contingent analysis, and exploratory versus confirmatory research (e.g., refs. 3-6). We use the more general terms--postdiction and prediction--to capture this important distinction.A common thread among epistemologies of science is that postdiction is characterized by the use of data to generate hypotheses about why something occurred, and prediction is characterized by the acquisition of data to test ideas about what will occur. In prediction, data are used to confront the possibility that the prediction is wrong. In postdiction, the data are already known and the postdiction is generated to explain why they occurred.Testing predictions is vital for establishing diagnostic evidence for explanatory claims. Testing predictions assesses the uncertainty of scientific models by observing how well the predictions account for new data. Generating postd...

show abstract

Section: Preregistration In Practicementioning

confidence: 99%

The preregistration revolution

Nosek

Ebersole

DeHaven

et al. 2018

Proc. Natl. Acad. Sci. U.S.A.

1,437

1,267

View full text Add to dashboard Cite

show abstract

“…Importantly, we now both agree on the crucial need for direct replication in social and personality psychology. They also emphasize open science and replicability reforms must be tailored to each research domain, something we have previously advocated for relationship science (Campbell, Loving, & LeBel, 2014) and others have done in other domains (e.g., industrial/organizational psychology, Kepes & McDaniel, 2013).…”

mentioning

confidence: 99%

Falsifiability is not optional.

LeBel

Berger

Campbell

et al. 2017

Journal of Personality and Social Psychology

Self Cite

View full text Add to dashboard Cite

show abstract

“…critique potentially hinders the goal of building a replicable relationship science because it presumes that published findings are both true and sufficient for understanding a given effect. Indeed, relationship science is unlikely to be immune to replicability challenges (Campbell, Loving, & LeBel, 2014;Cheung et al, 2016;LeBel, Campbell, & Loving, 2017). Therefore, the fact that a particular association between relationship quality (i.e., C) and X or relationship quality and Y has been published is not a sufficient reason to devalue additional tests of it.…”

Section: "Isn't It Just…?"mentioning

confidence: 99%

Solutions to the problems of incremental validity testing in relationship science

Wang¹,

Eastwick²

2020

Preprint

View full text Add to dashboard Cite

Incremental validity testing (i.e., testing whether a focal predictor is associated with an outcome above and beyond a covariate) is common (e.g., 57% of Personal Relationships articles in 2017), yet it is fraught with conceptual and statistical problems. First, researchers often use it to overemphasize the novelty or counterintuitiveness of findings, which hinders cumulative understanding. Second, incremental validity testing requires that the focal predictor and the covariate represent separate constructs; researchers risk committing the “jangle fallacy” without such evidence. Third, the most common approach to incremental validity testing (i.e., standard multiple regression, 88% of articles) inflates Type I error and can produce invalid conclusions. We also discuss the relevance of these issues to dyadic/longitudinal designs, and we offer concrete solutions.

show abstract

Enhancing Transparency of the Research Process to Increase Accuracy of Findings: A Guide for Relationship Researchers

Cited by 14 publications

References 53 publications

The preregistration revolution

The preregistration revolution

Falsifiability is not optional.

Solutions to the problems of incremental validity testing in relationship science

Contact Info

Product

Resources

About