Rating scales institutionalise a network of logical errors and conceptual problems in research practices: A rigorous analysis showing ways to tackle psychology’s crises

Abstract

This article explores in-depth the metatheoretical and methodological foundations on which rating scales—by their very conception, design and application—are built and traces their historical origins. It brings together independent lines of critique from different scholars and disciplines to map out the problem landscape, which centres on the failed distinction between psychology’s study phenomena (e.g., experiences, everyday constructs) and the means of their exploration (e.g., terms, data, scientific constructs)—psychologists’ cardinal error. Rigorous analyses reveal a dense network of 12 complexes of problematic concepts, misconceived assumptions and fallacies that support each other, making it difficult to be identified and recognised by those (unwittingly) relying on them (e.g., various forms of reductionism, logical errors of operationalism, constructification, naïve use of language, quantificationism, statisticism, result-based data generation, misconceived nomotheticism). Through the popularity of rating scales for efficient quantitative data generation, uncritically interpreted as psychological measurement, these problems have become institutionalised in a wide range of research practices and perpetuate psychology’s crises (e.g., replication, confidence, validation, generalizability). The article provides an in-depth understanding that is needed to get to the root of these problems, which preclude not just measurement but also the scientific exploration of psychology’s study phenomena and thus its development as a science. From each of the 12 problem complexes; specific theoretical concepts, methodologies and methods are derived as well as key directions of development. The analyses—based on three central axioms for transdisciplinary research on individuals, (1) complexity, (2) complementarity and (3) anthropogenicity—highlight that psychologists must (further) develop an explicit metatheory and unambiguous terminology as well as concepts and theories that conceive individuals as living beings, open self-organising systems with complementary phenomena and dynamic interrelations across their multi-layered systemic contexts—thus, theories not simply of elemental properties and structures but of processes, relations, dynamicity, subjectivity, emergence, catalysis and transformation. Philosophical and theoretical foundations of approaches suited for exploring these phenomena must be developed together with methods of data generation and methods of data analysis that are appropriately adapted to the peculiarities of psychologists’ study phenomena (e.g., intra-individual variation, momentariness, contextuality). Psychology can profit greatly from its unique position at the intersection of many other disciplines and can learn from their advancements to develop research practices that are suited to tackle its crises holistically.

What's wrong with rating scales? Psychology's replication and confidence crisis cannot be solved without transparency in data generation

“…To establish reliable and valid data sets and to analyse their empirical structures, statistics have become essential. Accordingly, current debates about problematic research practices (e.g., p ‐hacking, HARKing) and psychology's crises in replication, confidence, validation and generalisability (Andrade, 2021; Earp & Trafimow, 2015; Yarkoni, 2022) are focussed on data analysis (Uher, 2021d, 2022a, 2022b)—as are proposals for tackling them (e.g., pre‐registration, robust statistics; Nosek et al., 2015; Zwaan et al., 2017).…”

“…Applications of quantitative methods in psychology in themselves are increasingly questioned, such as regarding their underlying epistemologies and measurement theories (Barrett, 2018; Buntins et al., 2016; Michell, 1999; Tafreshi et al., 2016; Trendler, 2009, 2013; Uher, 2021c, 2022b; Westerman, 2014) or regarding the inadequacy of sample‐level statistics for individual‐level explorations (Lamiell, 2019; Molenaar & Campbell, 2009; Richters, 2021). Finally, psychology's neglected and insufficiently developed philosophical and theoretical foundations are identified as the root cause of its persistent crises (Danziger, 1985; Haig & Borsboom, 2008; Smedslund, 2016; Szollosi et al., 2020; Teo, 2018; Toomela, 2018; Uher, 2021b; Valsiner, 2019).…”

“…This also concerns the philosophical and theoretical foundations of research methods, especially of those used to generate data before these can be analysed (Uher, 2019, 2021a, 2022b; Valsiner, 2017; Wagoner & Valsiner, 2005). Indeed, open and ‘meta‐science’ initiatives focus on transparency in data analyses, such as in response coding and transformation, construct operationalisation and validity, as well as in the statistical tests, coefficients and parameters used (Flake & Fried, 2020; Hardwicke et al., 2022).…”

“…This article presents an introductory overview of selected key problems in the epistemological and methodological foundations of rating ‘scales’ that have been explored using the Transdisciplinary Philosophy‐of‐Science Paradigm for Research on Individuals (TPS‐Paradigm 1 ; for brief summaries see Uher, 2018a, 2021b, pp. 219–222, 2022b, pp. 4–5).…”

“…It provides overarching philosophical, metatheoretical and methodological frameworks that coherently build upon each other and into which established concepts from various disciplines have been systematically integrated and complemented by novel ones. These involve in particular (1) concepts of psyche, behaviour, language and contexts (e.g., Uher, 2013, 2016a, 2016b); (2) concepts and methodologies for taxonomising and comparing individual differences in various kinds of phenomena within and across populations (e.g., Uher, 2015a, 2015b, 2015c, 2015e, 2018b; Uher, Addessi, & Visalberghi, 2013), as well as (3) concepts and theories of data generation, quantification and measurement across the sciences (e.g., Uher, 2019, 2020b, 2022a) and in quantitative psychology and psychometrics (e.g., Uher, 2018a, 2021c, 2021d, 2022b; Uher & Visalberghi, 2016; Uher, Werner, & Gosselt, 2013). The TPS‐Paradigm's frameworks therefore provide strong conceptual foundations for scrutinising rating ‘scales’ and the common assumptions that they could enable psychological ‘measurement’ as will be shown now.…”