Construct validity and the validity of replication studies: A systematic review.

Abstract

Currently, there is little guidance for navigating measurement challenges that threaten construct validity in replication research. To identify common challenges and ultimately strengthen replication research, we conducted a systematic review of the measures used in the 100 original and replication studies from the Reproducibility Project: Psychology (Open Science Collaboration, 2015). Results indicate that it was common for scales used in the original studies to have little or no validity evidence. Our systematic review demonstrates and corroborates evidence that issues of construct validity are sorely neglected in original and replicated research. We identify four measurement challenges replicators are likely to face: a lack of essential measurement information, a lack of validity evidence, measurement differences, and translation. Next, we offer solutions for addressing these challenges that will improve measurement practices in original and replication research. Finally, we close with a discussion of the need to develop measurement methodologies for the next generation of replication research.

Measurement practices in large-scale replications: Insights from Many Labs 2.

“…We instituted quality control mechanisms throughout the coding process to facilitate transparency and ameliorate the subjective aspects of our review. First, the lead and senior authors on the paper (MS; JKF) adapted a coding key from previous research (Flake, Davidson, Wong, & Pek, 2019) and piloted it using a sample of seven effects. Once we finalized the coding sheet structure, two other researchers (SE; RL) were trained using three studies.…”

Limitations of the Sum-and-Alpha Approach to Measurement in Behavioral Research

“…When that connection is tenuous or non-existent, even consistent and repeated results can leave theoretical mechanisms unknown because it is unclear whether the variable in the statistical model is the same thing as the construct in the theoretical model. In effect, poor measurement obviates need for replication studies in the first place-if the original study failed to provide validity evidence for its measures, there is little purpose to pursue a replication because either result will not be informative for updating knowledge and understanding mechanisms of behavior (Flake et al, 2022).…”

“…Specifically, poor measurement confounds interpretations of replication studies (Flake et al, 2022). Imagine if a second research team tried to replicate the hypothetical depression intervention study discussed in the previous subsection and found that there were no differences between intervention and control groups.…”

“…If the target outcome is meaningless to begin with due to poor measurement and the replication follows the same protocol, then what difference does it make if the study replicates? Compounding this issue, Flake et al (2022) find that fewer than 10% of replication studies from the Reproducibility Project: Psychology (Open Science Collaboration, 2015; the largest reproducibility project to date) provided internal structure evidence for multiple-item scales.…”

A call for renewed attention to construct validity and measurement in psychopathology research

“…Borsboom, Mellenbergh, & van Heerden, 2004; Messick, 1987; Sechrest, 2005) that departs from classical approaches to construct validation. The relevant literature is complex and continues to develop to this day (Flake et al, 2022). Summarizing this work is beyond the scope of the current paper, in which I aim to bridge the divide between measurement theory and research on the etiology of psychopathology by focusing on pragmatic concerns surrounding the validity of putative vulnerabilities to psychopathology and their measurement.…”