The ideal experiment in physics must be conceptual, cunning, and conclusive. Adoption of these same standards in psychology has led to experiments that are uninformative and frivolous. We explain why we believe that psychology is fundamentally incompatible with hypothesis-driven theoretical science and conclude that this erodes the logic behind recent proposals to improve psychological research, such as stricter statistical standards, preregistration, and replication. The failure of psychology is not that it is somehow insufficiently scientific but rather that it makes inordinate use of methods that are a mismatch for the aspirations of researchers in the field, at the expense of valuable empirical research.
Marks & O’Connell (2021)’s claim that an even-handed review of the literature shows that social and economic background does not matter to educational outcomes in ad- vanced economies, relies on questionable scholarship and grave statistical errors. Much of the evidence against the relevance of social and economic background rests on a stub- born misinterpretation of research that estimates gain scores, where the effects of student background on educational outcomes are much smaller because they are residual effects. A lot of time is spent arguing that parental education and occupation are poor measures of student background but it is not recognized that this leads to attenuated effects and thus argues against their thesis. Finally, there is a lack of serious engagement with the literature: cherry-picked numbers, summaries of studies that are at odds with the conclu- sions of the original authors and even two cases where thought experiments are presented as empirical evidence. Research into inequity does at times lack rigor in measurement and analysis, and not accounting for heritable abilities can confound and exaggerate the association between student background or school composition and educational outcomes, but our calculations suggest that the effect is more than offset by attenuation due to measurement error.
Coarse data, mediation analyses, models that include intermediate outcomes and other purported statistical refinements tend to “residualize” the effect of social and economic background out of existence. We provide a brief overview of how overcorrection occurs and suggest how it may be avoided.
Scholars often hope to filter class, clout, poverty, deprivation, inequity and lack of opportunity from their analyses: factors that affect both how we act and what we achieve, and thereby muddle other associations in the data. In practice, adjustment relies on self- reported low-fidelity proxies such as occupation and education that capture but a fraction of the variance due to social and economic background and inflate the estimated effects of any correlated variable. When background is measured in a way that matches the research at hand, included in the analysis multidimensionally and not as a composite, and corrected for measurement error, it is not rare to see regression coefficients cut in half. We illustrate how poor measurement and poor models lead to residual confounding by social and economic background, review reliable alternatives to generic measures of socioeconomic status and demonstrate the effectiveness of statistical corrections for any error that remains.
The ideal experiment in physics must be conceptual, cunning and conclusive. Adoption of these same standards in psychology has led to experiments that are uninformative and frivolous. We explain why we believe that psychology is fundamentally incompatible with hypothesis-driven theoretical science and conclude that this erodes the logic behind recent proposals to improve psychological research, such as stricter statistical standards, preregistration and replication. The failure of psychology is not that it is somehow insufficiently scientific, but rather that it is inordinately fond of methods that are a mismatch for the aspirations of researchers in the field, at the expense of valuable empirical research.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.