Measurement versus prediction in the construction of patient-reported outcome questionnaires: can we have our cake and eat it?

Abstract

BackgroundTwo important goals when using questionnaires are (a) measurement: the questionnaire is constructed to assign numerical values that accurately represent the test taker’s attribute, and (b) prediction: the questionnaire is constructed to give an accurate forecast of an external criterion. Construction methods aimed at measurement prescribe that items should be reliable. In practice, this leads to questionnaires with high inter-item correlations. By contrast, construction methods aimed at prediction typically prescribe that items have a high correlation with the criterion and low inter-item correlations. The latter approach has often been said to produce a paradox concerning the relation between reliability and validity [1–3], because it is often assumed that good measurement is a prerequisite of good prediction.ObjectiveTo answer four questions: (1) Why are measurement-based methods suboptimal for questionnaires that are used for prediction? (2) How should one construct a questionnaire that is used for prediction? (3) Do questionnaire-construction methods that optimize measurement and prediction lead to the selection of different items in the questionnaire? (4) Is it possible to construct a questionnaire that can be used for both measurement and prediction?Illustrative exampleAn empirical data set consisting of scores of 242 respondents on questionnaire items measuring mental health is used to select items by means of two methods: a method that optimizes the predictive value of the scale (i.e., forecast a clinical diagnosis), and a method that optimizes the reliability of the scale. We show that for the two scales different sets of items are selected and that a scale constructed to meet the one goal does not show optimal performance with reference to the other goal.DiscussionThe answers are as follows: (1) Because measurement-based methods tend to maximize inter-item correlations by which predictive validity reduces. (2) Through selecting items that correlate highly with the criterion and lowly with the remaining items. (3) Yes, these methods may lead to different item selections. (4) For a single questionnaire: Yes, but it is problematic because reliability cannot be estimated accurately. For a test battery: Yes, but it is very costly. Implications for the construction of patient-reported outcome questionnaires are discussed.

Psychometric Properties of the Externalizing Spectrum Inventory: Replication and Extension across Clinical and Non-Clinical Samples

“…The apparent contradiction between the lack of support from the factor analyses and the present support from the criterion validity analyses, leads to the question which purpose the ESI can presently serve: Measurement or prediction? The current results suggest the latter, since accurate measurement of attributes requires high inter-item relations, while instruments aimed at prediction, require lower interitem correlations and higher correlations with an external criterion (Smits et al, 2018).…”

Methods for questionnaire design: a taxonomy linking procedures to test goals

“…The tradeoff among psychometric aspects is most easily shown for the internal and external methods as for both their central aspect may be quantified. By optimizing homogeneity, utilizing the internal method, instruments tend to show lower criterion validity, and by stressing criterion validity, externally developed instruments tend to show lower homogeneity (for mathematical proofs, and an empirical illustration, see, [ 84 ]). Similarly, the rational method produces instruments for which the reliability, content validity, and construct validity are not optimized, and it therefore seems reasonable to assume that they perform relatively poor on these psychometric qualities.…”

“…In addition, to determine the reliability of the scale, test–retest reliability coefficients are usually obtained [ 31 ]. In general, the external method tends to produce scales with low internal consistency coefficients [ 84 ], which is not surprising because heterogeneity instead of homogeneity is emphasized. Because the external method focuses on empirical relationships, it cannot be known if the resulting scale performs well on other criteria such as face validity and construct validity.…”

Some recommendations for developing multidimensional computerized adaptive tests for patient-reported outcomes

“…We did not discuss how to establish that it measures what it is supposed to measure (i.e., has adequate validity). This key quality criterion for any assessment measure needs to be demonstrated separately, for example, covering all relevant content [ 63 ], predicting external criteria [ 64 ], and fitting into the relations posed by the bank’s nomological net [ 65 ]. Third, item banks and CAT rest on the assumption that one or more latent variables determine the responses to the items (i.e., reflective measurement; [ 19 ]).…”