Critical examination of current response shift methods and proposal for advancing new methods

Abstract

Purpose This work is part of an international, interdisciplinary initiative to synthesize research on response shift in results of patient-reported outcome measures. The objective is to critically examine current response shift methods. We additionally propose advancing new methods that address the limitations of extant methods. Methods Based on literature reviews, this critical examination comprises design-based, qualitative, individualized, and preference-based methods, latent variable models, and other statistical methods. We critically appraised their definition, operationalization, the type of response shift they can detect, whether they can adjust for and explain response shift, their assumptions, and alternative explanations. Overall limitations requiring new methods were identified. Results We examined 11 methods that aim to operationalize response shift, by assessing change in the meaning of one’s self-evaluation. Six of these methods distinguish between change in observed measurements (observed change) and change in the construct that was intended to be measured (target change). The methods use either (sub)group-based or individual-level analysis, or a combination. All methods have underlying assumptions to be met and alternative explanations for the inferred response shift effects. We highlighted the need to address the interpretation of the results as response shift and proposed advancing new methods handling individual variation in change over time and multiple time points. Conclusion No single response shift method is optimal; each method has strengths and limitations. Additionally, extra steps need to be taken to correctly interpret the results. Advancing new methods and conducting computer simulation studies that compare methods are recommended to move response shift research forward.

Response shift in patient-reported outcomes: definition, theory, and a revised model

“…Various definitions and theories were proposed to integrate response shift in explaining change in self-reports [3,6,9,10]. Multiple methods were proposed to analyze response shift in PROM data [11,12]. Response shift was evidenced in various conditions [13,14].…”

“…In 2019, an international, interdisciplinary working group of 26 researchers, consisting of response shift experts, new investigators, and independent external experts was formed to achieve this synthesis [14]. They were divided in four teams [12,14,15], with the current team focusing on definition and theory.…”

Implications of response shift for micro-, meso-, and macro-level healthcare decision-making using results of patient-reported outcome measures

“…1 Potential response shift implications for use of PROM data at micro-, meso-, and macro-levels of healthcare decision-making situations, it is important to consider that some people may be more susceptible to response shifts than others. While various methods exist for evaluating the occurrence and magnitude of response, the "then-test" has been used most frequently (see Sébille et al for explanation of methods and limitations [8]). The "then-test" entails a retrospective assessment administered at follow-up asking for a re-evaluation of one's functioning at the time of the first assessment (pre-test) [17].…”

“…In these cases, response shift may not be identified even if it occurred. Where possible, we therefore recommend obtaining and analyzing longitudinal individual-level data to examine implications of response shift, and making statistical adjustments to account for the possibility of differential response shifts in some groups of people (e.g., by using latent variable methods [86], while taking limitations of current approaches into account [8]). Aggregating these adjusted PROM data for the purposes of quality improvement, performance monitoring, or accreditation will provide a sound basis for meso-level decision-making.…”

“…In so doing, we intend to raise awareness and provide guidance for PRO researchers who play an important role in informing healthcare providers, healthcare leaders, health policymakers, and regulators regarding the implications of response shift. This work is part of an international, interdisciplinary collaboration (see Appendix for the participating members) to synthesize the research on response shift to date [6][7][8].…”