Re-evaluating randomized clinical trials of psychological interventions: Impact of response shift on the interpretation of trial results

Abstract

Background Effectiveness of psychological treatment is often assessed using patient-reported health evaluations. However, comparison of such scores over time can be hampered due to a change in the meaning of self-evaluations, called ‘response shift’. Insight into the occurrence of response shift seems especially relevant in the context of psychological interventions, as they often purposefully intend to change patients’ frames of reference. Aims The overall aim is to gain insight into the general relevance of response shift for psychological health intervention research. Specifically, the aim is to re-analyse data of published randomized controlled trials (RCTs) investigating the effectiveness of psychological interventions targeting different health aspects, to assess (1) the occurrence of response shift, (2) the impact of response shift on interpretation of treatment effectiveness, and (3) the predictive role of clinical and background variables for detected response shift. Method We re-analysed data from RCTs on guided internet delivered cognitive behavioural treatment (CBT) for insomnia in the general population with and without elevated depressive symptoms, an RCT on meaning-centred group psychotherapy targeting personal meaning for cancer survivors, and an RCT on internet-based CBT treatment for persons with diabetes with elevated depressive symptoms. Structural equation modelling was used to test the three objectives. Results We found indications of response shift in the intervention groups of all analysed datasets. However, results were mixed, as response shift was also indicated in some of the control groups, albeit to a lesser extent or in opposite direction. Overall, the detected response shifts only marginally impacted trial results. Relations with selected clinical and background variables helped the interpretation of detected effects and their possible mechanisms. Conclusion This study showed that response shift effects can occur as a result of psychological health interventions. Response shift did not influence the overall interpretation of trial results, but provide insight into differential treatment effectiveness for specific symptoms and/or domains that can be clinically meaningful.

Response Shift After Cognitive Behavioral Therapy Targeting Severe Fatigue: Explorative Analysis of Three Randomized Controlled Trials

“…Put differently, while CBT induced a discrepancy between the observed scores and target construct, this did not affect the measured reduction in fatigue severity after CBT. This is in line with recent studies that found a limited impact of response shift on change in the target construct [22,54]. While the detected response shifts in this study were of a large magnitude, they occurred only on a single item per trial, thereby limiting their impact on overall fatigue severity, as measured by eight items.…”

“…In the Cancer-and Diabetes-trials, changes in some fatigue-related cognitions and perceptions showed small associations with the detected response shift on the item "I tire easily." Measures of changes in cognitions and perceptions might not be perfect operationalizations of changes in meaning and the method applied to investigate their relation is not well established yet (however, see also [22,54]). Nevertheless, these results may suggest that cognitions and perceptions could be part of the mechanisms underlying the detected response shift.…”

“…Response shift has been studied in patients who face a catalyst that triggers either a (temporal) deterioration in patients' health status (e.g., toxic cancer treatment or disease progression) or improvement in their health (e.g., intervention aimed at symptom relief). These studies provided evidence that response shift in the measurement of self-reported health outcomes, including fatigue, can occur [17][18][19][20][21] and that it may be triggered by an intervention such as CBT [22]. These studies also indicate that the occurrence of response shift, if not taken into account, can lead to biased conclusions -either over-or underestimation -about the health impact of the catalyst [19,20,23,24].…”

“…Based on response shift theory, a significant effect of a mechanism variable on an item affected by response shift is taken to indicate that change in the mechanism variable is a possible explanation for the detected response shift in that item. Because of the relative novelty of this approach to explore possible mechanisms of response shift, the results need to be interpreted in an exploratory fashion (however, see also [22,54] for applications of the same procedure). The effects of mechanism variables can be represented as Cohen's r, where values of 0.1, 0.3, and 0.5 are indicative of small, moderate, and large effects, respectively [51].…”

Listening to the elephant in the room: response-shift effects in clinical trials research

“…The 17 articles were predominantly secondary or post-hoc analyses (n = 12) [ 34 – 45 ], rather than explicitly designing the study to address response-shift effects (n = 5) [ 19 , 46 – 49 ] (Fig. 2 ).…”