Improving measurement in health education and health behavior research using item response modeling: comparison with the classical test theory approach

Abstract

This paper compares the approach and resultant outcomes of item response models (IRMs) and classical test theory (CTT). First, it reviews basic ideas of CTT, and compares them to the ideas about using IRMs introduced in an earlier paper. It then applies a comparison scheme based on the AERA/APA/NCME 'Standards for Educational and Psychological Tests' to compare the two approaches under three general headings: (i) choosing a model; (ii) evidence for reliability--incorporating reliability coefficients and measurement error--and (iii) evidence for validity--including evidence based on instrument content, response processes, internal structure, other variables and consequences. An example analysis of a self-efficacy (SE) scale for exercise is used to illustrate these comparisons. The investigation found that there were (i) aspects of the techniques and outcomes that were similar between the two approaches, (ii) aspects where the item response modeling approach contributes to instrument construction and evaluation beyond the classical approach and (iii) aspects of the analysis where the measurement models had little to do with the analysis or outcomes. There were no aspects where the classical approach contributed to instrument construction or evaluation beyond what could be done with the IRM approach. Finally, properties of the SE scale are summarized and recommendations made.

Confirmatory Factor Analysis of the Malay Version of the Confusion, Hubbub and Order Scale (CHAOS-6) among Myocardial Infarction Survivors in a Malaysian Cardiac Healthcare Facility

“…Future cross-validation studies should include more diverse and heterogeneous samples to exclude any homogeneity effects in our results. Certain low standardised factor loadings in the current study could be improved with the application of item response theory (IRT), a behaviour change assessment that uses a model-based approach to measure whether each item is fitted properly to the study population and culture (27). Our analyses indicate that CHAOS-6 reliably measure something, but we cannot yet state with certainty that what it measures is 'chaos'.…”

Rasch Analyses of the Activities-specific Balance Confidence Scale With Individuals 50 Years and Older With Lower-Limb Amputations

“…Rewording item 13 to only inquire about being bumped Through the use of Rasch modelling, the item and respondent characteristics may be estimated independent of each other. 33,38,41 This property of sample invariance allows for the measurement of people to be made independently of the items used, and the characteristics of the items to be evaluated independent of the respondents. 42 As a result, one can evaluate the scale's content representation relative to both the respondentsand the confidence continuum.…”

“…The likelihood ratio test compared the difference between the models' deviance parameters. 38 Deviance equals twice the log-likelihood and has a chi-square distribution with degrees of freedom equal to the difference in the number of parameters derived by the models. 38 The model with significantly less deviance and the fewest poor fitting items was used for further analyses.…”

“…38 Deviance equals twice the log-likelihood and has a chi-square distribution with degrees of freedom equal to the difference in the number of parameters derived by the models. 38 The model with significantly less deviance and the fewest poor fitting items was used for further analyses.Item difficulty, Redundancy, and Content gaps-Item difficulty, redundancy, and content gaps in the ABC-scale were assessed with a Rasch item-person map. The itemperson map locates both the items-by-threshold (i.e.…”

Use of Item Response Theory to Examine a Cardiovascular Health Knowledge Measure for Adolescents with Elevated Blood Pressure

“…IRT may be useful for designing a measure as well as examining the psychometric properties of an existing measure including reliability, construct validity, item difficulty, and item discrimination. In contrast to classical test theory (Hambleton & Jones, 1993;Wilson, Allen, & Li, 2006) IRT places item difficulties and person factor scores on the same metric, making it easier to establish which items target the construct of interest (e.g., health knowledge) and are appropriate for the given population. Furthermore, the placement of item difficulties and person factor scores on the same metric has implications for assessing change over time.…”