Understanding the exploratory/confirmatory data analysis continuum: Moving beyond the “replication crisis”.

Abstract

In light of the "replication crisis," some advocate for stricter standards and greater transparency in research methods. These efforts push toward a data analysis approach called "confirmatory data analysis" (CDA; see Wagenmakers et al., 2012). However, some (e.g., Baumeister, 2016;Goldin-Meadow, 2016) suggest that emphasizing CDA may restrict creativity and discovery. These scholars argued (sometimes inadvertently) for greater freedom to pursue "exploratory data analysis" (EDA; see Tukey, 1977). Ironically and unfortunately, many who push against stricter CDA standards do not realize EDA exists, or misunderstand the philosophy and proper tools for exploration. In this article, the meaning, tools, philosophy, and ethics associated with EDA, CDA, and a relatively unknown but important approach called "rough CDA" are clarified. Important distinctions are developed between EDA/rough CDA/CDA and other (some problematic) analysis activities including p-hacking, HARKing, and data mining, which are situated in a (graphical) framework that clarifies relationships and ethical boundaries with each. In short, the proper data analytic approach depends on (a) intentions and (b) transparency. Most psychological research is not at a maturity level to justify CDA; researchers have historically used tools mismatched to their research agenda. In the conclusion, recommendations are presented about how these typologies can be integrated into graduate training programs and how a cumulative research program can help psychology move beyond the replication crisis.

Speed–Accuracy Trade-Off? Not So Fast: Marginal Changes in Speed Have Inconsistent Relationships With Accuracy in Real-World Settings

“…In particular, we argue that our approach is reasonable for elucidating key facts about the SAT that we study here but we would not consider it a plausible generative model; indeed, the breadth of data here may require qualitatively different types of generative models. In being an approach focused not on testing models that may have generated the data but instead on key empirical relationships, we view it as being in the tradition of Tukey and others advocating for such exploratory approaches (Fife & Rodgers, 2021; Tukey et al, 1977). Our confidence in our approach’s ability to capture key features of the SAT is based on extensive simulation studies in the context of several different models for the joint distribution of time and accuracy, see SI.…”

Perceived Stress Mediates the Relationship Between Trait Mindfulness and Physical Symptoms of Stress: a Replication Study Using Structural Equation Modeling

“…Analyses were completed using a confirmatory data analysis approach to replicate prior findings (Fife & Rodgers, 2022 ). All hypotheses were addressed by first assessing psychometrics, univariate distributions, graphics, and assumptions of normality, linearity, and homoskedasticity (Fife, 2020 ).…”

“…Moreover, to account for all facets of mindfulness and covariates simultaneously, the present study uses structural equation modeling (SEM), a parsimonious analytic approach well matched to the research question. Finally, the present study helps advance the field by meeting the need for confirmatory data analysis as a way of moving beyond psychology’s replication crisis (Fife & Rodgers, 2022 ) and helping to “mind the hype” in mindfulness research (Van Dam et al, 2018 ). Based on both mindfulness stress buffering theory and prior empirical findings, we hypothesized that perceived stress would mediate the relationship between all facets of trait mindfulness (except observing) and physical symptoms, with non-judging showing the strongest effect.…”

Recruter de meilleurs échantillons en éducation

“…Pour illustrer ce point, considérons ce qui peut se produire lorsque les chercheurs recourent aux services de la plate-forme de travail en ligne Mechanical Turk fondée en 2005 par l'entreprise Amazon pour faire réaliser des tâches informatiques qui ne peuvent pas être automatisées (Buhrmester, Talaifar et Gosling, 2018) la recherche en sciences sociales, voir Camerer et al, 2018). Bien qu'il ne s'agisse pas du seul élément, la qualité de l'échantillon contribue inévitablement à la reproductibilité (Fife et Rodgers, 2021;Maxwell, Lau et Howard, 2015;Simmons et al, 2011).…”

“…En d'autres termes, si aucune étude n'est définitive, pourquoi utiliser la méthodologie la plus robuste et tenter de recruter le meilleur échantillon? Malheureusement, ce qui pourrait être appelé l'attentisme en matière de rigueur scientifique limite les progrès scientifiques en créant une accumulation d'études problématiques qui génère de la confusion (pour un examen des pratiques analytiques ou des devis problématiques, voirFife et Rodgers, 2021;Slavin et Cheung, 2017). Les…”