Ten steps toward a better personality science - how quality may be rewarded more in research evaluation

Abstract

In February 2020, the Personality and Diagnostics Chapter (DPPD) of the German Psychological Society (DGPs) tasked this group of authors with outlining what should be considered “good personality science”, as a positive vision of how to improve the credibility of research in the field. We argue in favor of writing consensus papers covering (1) shared, important research goals, (2) standardized use of terminology, (3) standardized measurement practices, and (4) shared views of the current state of theory and knowledge. All of these should help streamline the field considerably. We also argue in favor of (5) theory formalization, (6), pre-registration requirements for any confirmatory claims, (7) valuing replication attempts more (e.g., by reserving a quota of journal space for them), (8) planning for informative (e.g., well-powered) studies, and (9) making data, code, and materials open to the public by default. The current, quantity-based incentive structures in academia clearly stand in the way of implementing many of these practices, resulting in a research literature with sometimes questionable integrity and/or utility. As a solution, we propose a quality-based reward scheme that explicitly weights published research by its good science merits. Adoption of such a reward scheme may incur a steep decline in overall publication numbers, hopefully resulting in (a) an improved signal-to-noise ratio in the literature, and (b) more efficient allocation of resources (e.g., time) by researchers, who would be enabled to read more of what is being published, and to review each other’s work more carefully. Scientists need to be increasingly rewarded for doing good work, not just lots of work.

Replicability, Robustness, and Reproducibility in Psychological Science

“…These trends affect all areas of science but might be particularly acute in fields in which there is a lack of consensus about constructs, definitions, and theories (Leising et al, 2020). With incentives rewarding innovation, it is in researchers' self-interest to avoid using constructs and theories developed by others as illustrated by the aphorism "Psychologists treat theories like toothbrushes--no self-respecting person wants to use anyone else's" (Mischel, 2008).…”

Replicability, Robustness, and Reproducibility in Psychological Science

“…These trends that affect all areas of science might be particularly acute in fields, like psychology, in which there is a lack of consensus about constructs, definitions, and theories (Leising et al, 2020). With incentives rewarding innovation, it is in researchers' self-interest to avoid using constructs and theories developed by others as illustrated by the aphorism "Psychologists treat theories like toothbrushes--no self-respecting person wants to use anyone else's" (Mischel, 2008).…”

Against the “one method fits all data sets” philosophy for comparison studies in methodological research

“…We also see the previously described parallels between empirical and methodological research with respect to the pressure to find significant gains of a new treatment or method, and the risk of negative research findings ending up unpublished in a file drawer (Boulesteix et al, 2013). Just like the empirical sciences, we should therefore discuss both technical and systemic means and incentives to counter such issues (see, e.g., Leising et al, 2021). We believe that shifting the focus of methods comparisons away from finding an overall winner toward more differentiated results incorporating data set characteristics can add to this process in two ways.…”