Do the numbers speak for themselves? A critical analysis of procedural objectivity in psychotherapeutic efficacy research

Abstract

Psychotherapy research is known for its pursuit of evidence-based treatment (EBT). Psychotherapeutic efficacy is assessed by calculation of aggregated differences between pre treatment-and post treatment symptom levels. As this 'gold standard methodology' is regarded as 'procedurally objective', the efficacy number that results from the procedure is taken as a valid indicator of treatment efficacy. However, I argue that the assumption of procedural objectivity is not justified, as the methodology is build upon a problematic numerical basis. I use an empirical case study to show (1) how measurement problems practically occur in the first step of data collection, i.e. in individual symptom measurement. These problems have been discussed and acknowledged for decades, but still measurement is regarded as the best epistemic means to gain evidence on psychotherapeutic efficacy. Therefore, I show (2) how initial measurement problems are overlooked in the remainder of the methodological procedure, which harms the 'evidence-base' of psychotherapeutic EBT. Via this applied analysis, I exhibit concerns that are increasingly raised in the literature in an empirical way, to emphasize the need for a non-idealized consideration of the 'gold standard methodology' as a means towards its clinical end.

To manualize, or not to manualize: Is that still the question? A systematic review of empirical evidence for manual superiority in psychological treatment

“…Indeed, it arrests researchers’ and stakeholders’ interest at the level of the manual and at the level of the sample, which is neither an epistemically nor clinically fruitful imperative (cf. Desmet, ; Truijens, ). Epistemic progress seems to require that the researcher does not embrace manual‐use a priori and generally but carefully scrutinizes when, why and how specific techniques can be utilized in clinical practice.…”

Paradoxical outcomes in psychotherapy: Theoretical perspectives, research agenda and practice implications

“…Studies in which individuals are invited to describe their experience of completing symptom measures used in therapy research have found that participants describe items as irrelevant or hard to follow, or interpret items in ways that are different from the meanings intended by the test designer (see, for example, Galasiński & Kozłowska, 2013;Paz et al, 2020;Truijens et al, 2019b). There are many factors that influence the ways that respondents approach the task of answering questionnaire items (McClimans, 2010;McLeod, 2001;Truijens, 2017;Truijens et al, 2019a). For example, a person may understand specific words, such as 'always' or 'sometimes' in an idiosyncratic or culturally determined manner, they may want to reply 'it depends' rather than provide a general answer, or they may be influenced by what they believe the purpose of the measure to be or their feelings about the individual administering it.…”

Validity Beyond Measurement: Why Psychometric Validity Is Insufficient for Valid Psychotherapy Research

“…This cross-validation or triangulation indeed indicates a reduction of initial depressive symptoms, and therefore supports a positive conclusion regarding treatment success. Note that it is somewhat simplistic to reach this conclusion based on a single individual pre-post difference, yet it still functions as an illustration of common methodological reasoning in psychotherapeutic research (see Truijens, 2017 , for a discussion of this line of reasoning in psychotherapeutic efficacy methodology).…”

“…However, it is not evident to reach a sound and clear conclusion based on this number; to make sense of this increase, the idiosyncratic information should be taken into account to find out whether the increase is related to depressive symptoms or to, for example, regular life stressors (cf. Truijens, 2017 ).…”