Using Classical Test Theory, Item Response Theory, and Rasch Measurement Theory to Evaluate Patient-Reported Outcome Measures: A Comparison of Worked Examples

Abstract

Selection of a psychometric approach depends on many factors. Researchers should justify their evaluation method and consider the intended audience. If the instrument is being developed for descriptive purposes and on a restricted budget, a cursory examination of the CTT-based psychometric properties may be all that is possible. In a high-stakes situation, such as the development of a patient-reported outcome instrument for consideration in pharmaceutical labeling, however, a thorough psychometric evaluation including IRT or RMT should be considered, with final item-level decisions made on the basis of both quantitative and qualitative results.

Psychometric validation of the Coronary Revascularisation Outcome Questionnaire (CROQv2) in the context of the NHS Coronary Revascularisation PROMs Pilot

“…Classical Test Theory (CTT), [11][12][13][14][15] the traditional psychometric approach, is the most dominant paradigm in the development of PROMs. 16 Within CCT, well-established methods and criteria are applied to indicate that the concept, that is, represented by the PROM is clear and well understood; that the content is relevant to the patient group concerned; that the psychometric properties (acceptability, reliability, validity, responsiveness) are adequate; and that the scaling structure is justified. [17][18][19] The psychometric properties of a PROM are sample and context dependent within the CTT paradigm.…”

Centralization and delegation practices in family versus non-family SMEs: a Rasch analysis

“…In fact, in head-to-head comparisons many studies have found Rasch to provide similar or superior results to those provided by CTT (cf. Engelhard and Wang 2014; Petrillo et al 2015).…”

Validation of the Dutch version of the Swallowing Quality-of-Life Questionnaire (DSWAL-QoL) and the adjusted DSWAL-QoL (aDSWAL-QoL) using item analysis with the Rasch model: a pilot study

“…The CTT-psychometric assessment included an examination of internal consistency based on Cronbach’s alpha, test-retest reliability via the intraclass correlation coefficient (ICC), and/or construct validity based on principal component analysis (PCA) techniques [7, 11, 13]. Some drawbacks related to CTT methods are recognized [14, 15], such as test and sample dependence [14–16] and the assumption of equal weight for all of the items even if there is a difference in the level of difficulty [16]. The scale’s total sum score is based on ordinal values and the standard error of measurement is assumed to be constant [10, 15], in contrast to the Rasch methodology.…”

“…Some drawbacks related to CTT methods are recognized [14, 15], such as test and sample dependence [14–16] and the assumption of equal weight for all of the items even if there is a difference in the level of difficulty [16]. The scale’s total sum score is based on ordinal values and the standard error of measurement is assumed to be constant [10, 15], in contrast to the Rasch methodology.…”