Identifying meaningful change on PROMIS short forms in cancer patients: a comparison of item response theory and classic test theory frameworks

Abstract

Background This study compares classical test theory and item response theory frameworks to determine reliable change. Reliable change followed by anchoring to the change in categorically distinct responses on a criterion measure is a useful method to detect meaningful change on a target measure. Methods Adult cancer patients were recruited from five cancer centers. Baseline and follow-up assessments at 6 weeks were administered. We investigated short forms derived from PROMIS® item banks on anxiety, depression, fatigue, pain intensity, pain interference, and sleep disturbance. We detected reliable change using reliable change index (RCI). We derived the T-scores corresponding to the RCI calculated under IRT and CTT frameworks using PROMIS® short forms. For changes that were reliable, meaningful change was identified using patient-reported change in PRO-CTCAE by at least one level. For both CTT and IRT approaches, we applied one-sided tests to detect reliable improvement or worsening using RCI. We compared the percentages of patients with reliable change and reliable/meaningful change. Results The amount of change in T score corresponding to RCICTT of 1.65 ranged from 5.1 to 9.2 depending on domains. The amount of change corresponding to RCIIRT of 1.65 varied across the score range, and the minimum change ranged from 3.0 to 8.2 depending on domains. Across domains, the RCICTT and RCIIRT classified 80% to 98% of the patients consistently. When there was disagreement, the RCIIRT tended to identify more patients as having reliably changed compared to RCICTT if scores at both timepoints were in the range of 43 to 78 in anxiety, 45 to 70 in depression, 38 to 80 in fatigue, 35 to 78 in sleep disturbance, and 48 to 74 in pain interference, due to smaller standard errors in these ranges using the IRT method. The CTT method found more changes compared to IRT for the pain intensity domain that was shorter in length. Using RCICTT, 22% to 66% had reliable change in either direction depending on domains, and among these patients, 62% to 83% had meaningful change. Using RCIIRT, 37% to 68% had reliable change in either direction, and among these patients, 62% to 81% had meaningful change. Conclusion Applying the two-step criteria demonstrated in this study, we determined how much change is needed to declare reliable change at different levels of baseline scores. We offer reference values for percentage of patients who meaningfully change for investigators using the PROMIS instruments in oncology.

Introduction to the special section: “Methodologies and considerations for meaningful change”

“…The team combined the use of a longitudinal graded response model with a transition item to measure latent change. The method produced tighter estimates of meaningful change when compared to traditional methods, with the methods overlapping most when the proportion of responders was [23] N/A Individual Change over time N/A Griffiths et al [24] Meaningful change Group, Individual Change over time Minimal Ho et al [33] Distribution-based Individual, Group Change over time N/A Jones et al [21] Meaningful change Individual Change over time Not specified Lee et al [32] Both Individual Change over time Minimal Li [18] Distribution-based Individual Change over time N/A Peipert et al [30] Distribution-based Individual Change over time N/A Poon et al [29] Meaningful change Individual Change over time (hypothetical) Minimal Qin et al [27] Meaningful change Individual Change over time Not specified Smit et al [16] Both Individual Change over time Meaningful a Wyrwich & Norman [22] Meaningful change General General General Wyrwich et al [19] Meaningful change Individual Change over time (hypothetical) Meaningful b about 50% of participants. Extensions of this approach show promise for a range of applications [25,26].…”

“…In addition, the paper has two letters attached to it in this same issue, which discuss the interpretation of the attached statistical significance level and the applicability of the index to individual change classification, which are also of interest for other indices and their interpretation. The second paper [32] focuses also on a version of the reliable change index and compares its use based on classical test theory and item response theory. Classical test theory assumes measurement error is constant across the scale range, but item response theory relaxes this assumption.…”

Introduction to the special section "Methodologies and Considerations for Meaningful Change"

“…In addition, the paper has two letters attached to it in this same issue, which discuss the interpretation of the attached statistical significance level and the applicability of the index to individual change classification, which are also of interest for other indices and their interpretation. The second paper (33), focuses also on a version of the reliable change index and compares its use based on classical test theory and item response theory. Classical test theory assumes measurement error is constant across the scale range, but item response theory relaxes this assumption.…”

Development and validation of the biosafety incident response competence scale for clinical nursing staff

“…And we removed the highest factor load less than 0.4, factor load across two or more factors and the difference less than 0.2, and the number of common factors included items less than 3 [21]. According to the delete criteria, items 8, 9,10,14,21,26,27,31,34,35,36,37,38,39,40,41 were deleted, and 4 common factors were extracted. The cumulative contribution of variance accounted for 73.427%.…”

“…A conceptual model was constructed based on literature review, qualitative interviews, group meetings, and Delphi consultation. The research group used the Classic Test Theory (CTT) as a guide to screen items using critical ratio, discrete trend, correlation coe cient, factor loading and Cronbach's α coe cient [27]. On the basis of the conceptual model, the items of the scale were screened through two rounds of questionnaire surveys, the structure of the scale was explored, and the reliability and validity of the scale were tested.…”