A Skeptical Eye on Psi

Wagenmakers, Eric‐Jan; Wetzels, Ruud; Borsboom, Denny; Kievit, Rogier A.; Maas, Han L. J. van der

doi:10.31234/osf.io/syq2h

Cited by 67 publications

(82 citation statements)

References 54 publications

(55 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…found, authors may have been more inclined to highlight this finding as a focal result. These two possibilities may be disambiguated by preregistration (Wagenmakers et al, 2012).…”

Section: Methodological Moderatorsmentioning

confidence: 99%

Relationship between the Implicit Association Test and intergroup behavior: A meta-analysis

Kurdi¹,

Seitchik²,

Axt³

et al. 2018

Preprint

View full text Add to dashboard Cite

Using data from 217 research reports (N = 36,071, compared to 3,471 and 5,433 in previous meta-analyses), this meta-analysis investigated the conceptual and methodological conditions under which Implicit Association Tests (IATs) measuring attitudes, stereotypes, and identity correlate with criterion measures of intergroup behavior. We found significant implicit–criterion correlations (ICCs) and explicit–criterion correlations (ECCs), with unique contributions of implicit (β = .14) and explicit measures (β = .11) revealed by structural equation modeling. ICCs were found to be highly heterogeneous, making moderator analyses necessary. Basic study features or conceptual variables did not account for any heterogeneity: Unlike explicit measures, implicit measures predicted for all target groups and types of behavior, and implicit, but not explicit, measures were equally associated with behaviors varying in controllability and conscious awareness. However, ICCs differed greatly by methodological features: Studies with a declared focus on ICCs, standard IATs rather than variants, high-polarity attributes, behaviors measured in a relative (two categories present) rather than absolute manner (single category present), and high implicit–criterion correspondence (k = 13) produced a mean ICC of r = .37. Studies scoring low on these variables (k = 6) produced an ICC of r = .02. Examination of methodological properties—a novelty of this meta-analysis—revealed that most studies were vastly underpowered and analytic strategies regularly ignored measurement error. Recommendations, along with online applications for calculating statistical power and internal consistency (http://www.benedekkurdi.com/#iat), are provided to improve future studies on the implicit–criterion relationship. Open materials are available under https://osf.io/47xw8/.

show abstract

“…found, authors may have been more inclined to highlight this finding as a focal result. These two possibilities may be disambiguated by preregistration (Wagenmakers et al, 2012).…”

Section: Methodological Moderatorsmentioning

confidence: 99%

Relationship between the Implicit Association Test and intergroup behavior: A meta-analysis

Kurdi¹,

Seitchik²,

Axt³

et al. 2018

Preprint

View full text Add to dashboard Cite

show abstract

“…Third, advances in statistical methodology now allow for explicit modeling of dependencies among effect sizes extracted from the same study (Hedges, Tipton, & Johnson, 2010) as well as for the appropriate treatment of measurement error (Westfall & Yarkoni, 2016). Fourth, recent developments in the way psychological science is conducted, along with an increased focus on the transparency of statistical analyses, the replicability of results, and the standards for what constitutes good statistical evidence (e.g., Wagenmakers, Wetzels, Borsboom, van der Maas, & Kievit, 2012) also add to the need for a new review. The recent push toward open science makes it a particularly fitting time to produce a meta-analytic database that is sufficiently rich to be useful for additional investigation, and the preparation of this analy-sis was guided by that imperative.…”

Section: Relationship Between the Implicit Association Test And Intermentioning

confidence: 99%

Relationship between the Implicit Association Test and intergroup behavior: A meta-analysis.

Kurdi¹,

Seitchik²,

Axt³

et al. 2019

American Psychologist

322

378

View full text Add to dashboard Cite

Using data from 217 research reports (N = 36,071, compared to 3,471 and 5,433 in previous meta-analyses), this meta-analysis investigated the conceptual and methodological conditions under which Implicit Association Tests (IATs) measuring attitudes, stereotypes, and identity correlate with criterion measures of intergroup behavior. We found significant implicit-criterion correlations (ICCs) and explicit-criterion correlations (ECCs), with unique contributions of implicit (β = .14) and explicit measures (β = .11) revealed by structural equation modeling. ICCs were found to be highly heterogeneous, making moderator analyses necessary. Basic study features or conceptual variables did not account for any heterogeneity: Unlike explicit measures, implicit measures predicted for all target groups and types of behavior, and implicit, but not explicit, measures were equally associated with behaviors varying in controllability and conscious awareness. However, ICCs differed greatly by methodological features: Studies with a declared focus on ICCs, standard IATs rather than variants, high-polarity attributes, behaviors measured in a relative (two categories present) rather than absolute manner (single category present), and high implicit-criterion correspondence (k = 13) produced a mean ICC of r = .37. Studies scoring low on these variables (k = 6) produced an ICC of r = .02. Examination of methodological properties-a novelty of this meta-analysis-revealed that most studies were vastly underpowered and analytic strategies regularly ignored measurement error. Recommendations, along with online applications for calculating statistical power and internal consistency (http://www.benedekkurdi.com/#iat), are provided to improve future studies on the implicitcriterion relationship. Open materials are available under https://osf.io/47xw8/.

show abstract

“…Perhaps one of the most important steps to take in increasing the robustness and integrity of the research in infancy and developmental research is to preregister all studies and to clearly delineate confirmatory and exploratory research (Wagenmakers et al, 2012;Chambers & Munafò, 2013). Preregistration is similar to writing a proposal for a dissertation or grant such that all hypotheses are stated, power of the design is calculated, appropriate sample is selected, measurement is detailed, analyses are pre-planned, and all that is left is to gather the data based on these specifications.…”

Section: Best Practices For Future Researchmentioning

confidence: 99%

An overview of issues in infant and developmental research for the creation of robust and replicable science

Davis‐Kean

Ellis

2019

Infant Behavior and Development

View full text Add to dashboard Cite

In the last few years, the field of psychology has been challenged with a crisis in the rigor and reproducibility of the science. The focus of these issues has primarily been in social, cognitive, and cognitive neuroscience psychology, however, the area of developmental research is not immune to these issues. This paper provides an overview of the "replication crisis" and the choices made by researchers that are often not noted in methods, thus making the replication of studies more difficult. In this review we discuss issues of researcher flexibility in the data design and selection of sample size, collection, and analysis stages of research. In each of these areas we address examples of bias and how developmental researchers can address these issues in their own research.Robust and Replicable Science 4 they were discussed at a major psychological conference. As a result, a call for action was put forward to make psychological science more rigorous and the findings more replicable across all areas of psychology. Most of the examples provided at the meeting were related to the cognitive, social, and neuroscience areas of psychology where issues of small samples (i.e., low power) and lack of multiple comparison corrections are relatively frequent in publications. However, when the examples were presented at the meeting, developmentalists in the room recognized the similarities in their own research practices, and that we were not immune to the issues of research replicability and rigorous practices (see Munafò et al., 2017).

show abstract

A Skeptical Eye on Psi

Cited by 67 publications

References 54 publications

Relationship between the Implicit Association Test and intergroup behavior: A meta-analysis

Relationship between the Implicit Association Test and intergroup behavior: A meta-analysis

Relationship between the Implicit Association Test and intergroup behavior: A meta-analysis.

An overview of issues in infant and developmental research for the creation of robust and replicable science

Contact Info

Product

Resources

About