Optimizing Research Output: How Can Psychological Research Methods Be Improved?

Abstract

Recent evidence suggests that research practices in psychology and many other disciplines are far less effective than previously assumed, which has led to what has been called a “crisis of confidence” in psychological research (e.g., Pashler & Wagenmakers 2012). In response to the perceived crisis, standard research practices have come under intense scrutiny, and various changes have been suggested to improve them. The burgeoning field of metascience seeks to use standard quantitative data-gathering and modeling techniques to understand the reasons for inefficiency, to assess the likely effects of suggested changes, and ultimately to tell psychologists how to do better science. We review the pros and cons of suggested changes, highlighting the many complex research trade-offs that must be addressed to identify better methods. Expected final online publication date for the Annual Review of Psychology, Volume 73 is January 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

On doing multi-act arithmetic: A multitrait-multimethod approach of performance dimensions in integrated multitasking

“…In order to construct a test with excellent psychometric characteristics, one has to consider three aspects, the principles of measurement theory ( Lord and Novick, 1968 ; Lienert, 1969 ) the principles of chronometric-design theory ( Miller and Ulrich, 2013 , 2021 , 2022 ), and the principles of concept-generalizability theory ( Cronbach and Meehl, 1955 ; Stroebe and Strack, 2014 , pp. 61–63).…”

Preregistration: the good, the bad, and the confusing

“…Discussions that focus on replicability (Hoffmann et al 2021) may divert attention to issues that are important but peripheral to the issue of credibility. Isn't replication failure inevitable given the vagaries of reproducing statistically variable effects Miller 2020, Miller andUlrich 2022)? Should we be talking of exact (or direct) replication that challenges data robustness, or conceptual replication that challenges theoretical implications (Nosek and Errington 2020)?…”

Overcoming the phenomenological Perpetuum mobile in clinical cognitive neuroscience for the benefit of replicability in research and the societal view on mental disorders

“…Compared to the “#Manylabs” strategy, a pre-registration and a registered report can at least partly circumvent the problem imposed by regional and country differences in diagnosing mental disorders and the use of the diagnosis as a starting point for cognitive neuroscience research because the pre-registered study can be conducted at only one study site. However, recent analyses have shown that this strategy is not without problems ( Ulrich and Miller, 2020 ; Miller and Ulrich, 2022 ), and these problems are particularly central when it comes to cognitive neuroscience research in mental health:…”

“…A central problem that pre-registrations and registered reports cannot overcome refers to the “base rate” of phenomena. The base rate reflects the probability that a sought-after effect is truly present in the population ( Miller and Ulrich, 2022 ). The base rate is not an aspect of a specific study but of the entire research area in which the study is conducted and has a tremendous effect on the probability of obtaining a (non-replicable) false positive result ( Miller and Ulrich, 2022 ).…”

“…The base rate reflects the probability that a sought-after effect is truly present in the population ( Miller and Ulrich, 2022 ). The base rate is not an aspect of a specific study but of the entire research area in which the study is conducted and has a tremendous effect on the probability of obtaining a (non-replicable) false positive result ( Miller and Ulrich, 2022 ). Notably, the base rate is high in fields with a solid theoretical and empirical background.…”