An Application of the Three-Step Test-Interview (TSTI) in the Validation of the Relational Depth Frequency Scale

Abstract

The main objective of this study is to evaluate the utility of a new qualitative scale development methodology—Three-Step Test-Interview (TSTI)—in its first application in the validation of a psychotherapy scale: The Relational Depth Frequency Scale (RDFS). The TSTI is a cognitive pretesting method designed to uncover potential problems in scale construction. The RDFS is a six-item unidimensional scale of in-depth therapeutic relating, designed for use in large-scale outcome studies. Following the creation of an item pool and “expert ratings,” a purposive sample of four therapists and four clients (five females, three males, mean age: 49 years) was recruited to take part in the TSTI with the view to refine the original 36-item RDFS prior to psychometric exploration. Structured observations pointed to problems in test-takers’ patterns of responses in relation to theoretical knowledge of the relational depth construct. Issues uncovered and addressed included some misinterpretations of instructions and items, redundant content, double-barreled items, and test-takers’ reactions to intimate content wording. The method supported the refinement of the RDFS including amendment to its instructions and the removal of problematic items. TSTI results produced knowledge on the scale which could not be captured with statistical methods.

The Patient-Perceived Helpfulness of Measures Scale: Development and Validation of a Scale to Assess the Helpfulness of Using Measures in Psychological Treatment

“…Convergence with single-survey items was re-assessed in the new sample. In addition, we assessed measurement invariance across measures, convergent validity with a measure of psychotherapy satisfaction, and divergent validity with a measure of social desirability (Di Malta, et al, 2023). Reliability was assessed using both internal consistency and test–retest reliability with a sub-sample of participants after 1 month (DeVellis & Thorpe, 2021).…”

Extended validation of the Relational Depth Frequency Scale: Retest reliability, divergent validity, criterion validity with psychotherapy satisfaction, and measurement invariance in UK‐ and US‐stratified samples

“…The Relational Depth Frequency Scale (RDFS) is a short unidimensional self‐report measure developed to assess the frequency of RD over the course of psychotherapy (Di Malta, Evans, et al, 2020). The RDFS was developed based on standard procedures of scale development (DeVellis & Thorpe, 2021) and cognitive pretesting methods with patients and psychotherapists (Di Malta, Cooper, et al, 2020). The RDFS was psychometrically tested using confirmatory factor analysis (CFA) and Rasch analyses in online samples of psychotherapists and patients in the United Kingdom (Di Malta, Evans, et al, 2020).…”

“…The total score for the RDFS is summed and ranging from six to 30 with a higher score meaning more frequent RD, and no specified cut‐off scores. The RDFS has shown excellent internal consistency ( α > 0.90) and validity in online samples of therapists and patients (Di Malta, Cooper, et al, 2020; Di Malta, Evans, et al, 2020), and in a clinical sample of psychotherapy dyads (Zarzycka et al, 2022). In this study, the RDFS' internal consistency was Cronbach's α > 0.90 in the UK and US samples.…”

A Pluralistic Perspective on Research in Psychotherapy: Harnessing Passion, Difference and Dialogue to Promote Justice and Relevance

“…An important strand of that effort has been the development of a research base that would identify evidence to facilitate the development of pluralistic practice; conducting studies on the process and outcomes of pluralistic therapy itself (e.g. Cooper et al, 2015 ; Di Malta et al, 2020 ). The intention has been to build an approach to therapy that is research-informed rather than research-directed ( Hanley and Winter, 2016 ), based on the principle that effective practice draws on multiple ways of knowing: ethical, personal, theoretical, cultural and scientific ( McLeod, 2016 ).…”