Publish less, read more

Phaf, R. Hans

doi:10.1177/0959354319898250

Cited by 37 publications

(45 citation statements)

References 80 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As Cummins put it: "In psychology, we are overwhelmed with things to explain, and somewhat underwhelmed by things to explain them with" (Cummins, 2000). For this reason, many scholars have argued that psychology's attention to statistics and replicability has distracted from a problem that runs much deeper: a crisis of theory (Borsboom, 2013;Borsboom et al, 2020;Cummins, 2000;Guest & Martin, 2020;Haslbeck et al, 2019;Meehl, 1990b;Muthukrishna & Henrich, 2019;Phaf, 2020;Robinaugh, Haslbeck, et al, 2019;Smaldino, 2019;Szollosi et al, 2019;Vaidyanathan et al, 2015;Van Rooij & Baggio, 2020).…”

Section: Data Rich and Theory Poormentioning

confidence: 99%

Lack of theory building and testing impedes progress in the factor and network literature

Fried¹

2020

Preprint

113

138

View full text Add to dashboard Cite

The last decade has brought reforms to improve methodological practices, with the goal to increase the reliability and replicability of effects. However, explanations of effects remain scarce, and a growing chorus of scholars argues that the replicability crisis has distracted from a crisis of theory. In the same decade, the empirical literature using factor and network models has grown rapidly. I discuss three ways in which this literature falls short of theory building and testing. First, statistical and theoretical models are conflated, leading to invalid inferences such as the existence of psychological constructs based on factor models, or recommendations for clinical interventions based on network models. I demonstrate this inferential gap in a simulation study on statistical equivalence: excellent model fit does little to corroborate a theory, regardless of quality or quantity of data. Second, researchers fail to explicate theories about psychological constructs, but use implicit causal beliefs to guide inferences. These latent theories have led to problematic best practices in psychological research where inferences are drawn based on one specific causal model that is assumed, but not explicated. Third, explicated theories are often weak theories: narrative and imprecise descriptions vulnerable to hidden assumptions and unknowns. They fail to make clear predictions, and it remains unclear whether statistical effects corroborate such theories or not. Weak theories are immune to refutation or revision. I argue that these three challenges to theory building and testing are common and harmful, and impede theory formation, failure, and reform. A renewed focus on theoretical psychology and formal models offers a way forward.

show abstract

Section: Data Rich and Theory Poormentioning

confidence: 99%

Lack of theory building and testing impedes progress in the factor and network literature

Fried¹

2020

Preprint

113

138

View full text Add to dashboard Cite

show abstract

“…Although ensuring replicability, transparency, and openness are key to a healthier science (Chambers, 2019;Cohen, 2017;Moshontz et al, 2018;Nosek et al, 2018;Phaf, 2020;Poldrack et al, 2017;Zwaan, Etz, Lucas, & Donnellan, 2017), improving scientific inferences and building a cumulative science will require more. Psychology will benefit from diversifying its methods or the means through which data are collected and analyzed (e.g., Vazire, 2018).…”

Section: Diverse Methodsmentioning

confidence: 99%

“…When we look within psychology journals, it appears as though experiments have taken on a life of their own: 9 out of 10 articles in the Journal of Cognitive Neuroscience used experiments, PROMISES AND PERILS OF EXPERIMENTATION 13 though the proportions were smaller for Psychological Science (6 out of 10) and the Journal of Personality and Social Psychology (5 out of 9). Even recent large-scale multi-site replication projects (e.g., Many Labs Projects) seem to be doubling down on experimentation, presumably assuming that after identifying which experimental effects can(not) be replicated (see Phaf, 2020), the field will eventually regain its lost scientific respectability and credibility (Aschwanden, 2018;Nelson, Simmons, & Simonsohn, 2018). Ensuring replicability is crucial (but see Lewandowsky & Oberauer, 2020;Smaldino & McElreath, 2016), but what we suggest instead is the need to look beyond replicating experimental effects and consider approaches other than experimentation.…”

Section: Diverse Methodsmentioning

confidence: 99%

Promises and perils of experimentation: The mutual internal validity problem

Lin¹,

Werner²,

Inzlicht³

2020

Preprint

View full text Add to dashboard Cite

Experimentation is the cornerstone of the scientific method. Decades of experimental work in psychology, however, has yet to produce reliable and cumulative knowledge. We argue psychology as a discipline not only overuses experimentation, but too often uses it as a theatrical storytelling device. Rather than use experiments to facilitate valid scientific inference, psychologists cleverly design experiments to elicit the very effects they wish to observe, ignoring how these effects may or may not cohere with bodies of knowledge within psychology and across disciplines. This approach leads to ungeneralizable results and an unwillingness to accept and learn from failed, but potentially informative experiments. To make real progress toward understanding human behavior and the mind, psychology needs to embrace big-interdisciplinary (big-I) triangulation, which requires integrating theories, methods, and approaches from diverse disciplines and sources.

show abstract

“…A number of suggestions have been advanced to improve the quality of replications. Phaf (2020) suggested that experimental work should always be complemented by thorough theoretical analyses. In the case of unsuccessful replications, this would allow for the discovery of potentially crucial (and as yet unexamined) factors that may explain the result.…”

Section: Replication Level: Group Effectsmentioning

confidence: 99%