Rating scales and Rasch measurement

Abstract

Assessments with ratings in ordered categories have become ubiquitous in health, biological and social sciences. Ratings are used when a measuring instrument of the kind found in the natural sciences is not available to assess some property in terms of degree - for example, greater or smaller, better or worse, or stronger or weaker. The handling of ratings has ranged from the very elementary to the highly sophisticated. In an elementary form, and assumed in classical test theory, the ratings are scored with successive integers and treated as measurements; in a sophisticated form, and used in modern test theory, the ratings are characterized by probabilistic response models with parameters for persons and the rating categories. Within modern test theory, two paradigms, similar in many details but incompatible on crucial points, have emerged. For the purposes of this article, these are termed the statistical modeling and experimental measurement paradigms. Rather than reviewing a compendium of available methods and models for analyzing ratings in detail, the article focuses on the incompatible differences between these two paradigms, with implications for choice of model and inferences. It shows that the differences have implications for different roles for substantive researchers and psychometricians in designing instruments with rating scales. To illustrate these differences, an example is provided.

Toward a Standardized Reporting of Outcomes in Hand Osteoarthritis: Developing a Common Metric of Outcome Measures Commonly Used to Assess Functioning

“…This user-friendly and inclusive process toward achieving standardized reporting of outcome measures follows directly from the raw score sufficiency. Raw score sufficiency, which is an attribute of the Rasch measurement model, implies that the total score across a set of items provides all the information needed to estimate the person's ability on the metric of functioning (47). Therefore, a clinician only needs to calculate the total score, look up the corresponding Rasch-transformed score on the reference metric for each scale, (Table 4) and thereby receive an invariant, interval-scaled score that is comparable with similar scales used in hand OA.…”

Psychometric properties of the Epistemological Development in Teaching Learning Questionnaire (EDTLQ): An inventory to measure higher order epistemological development

“…Andrich (2011) points out a large difference between the traditional IRT paradigm and the Rasch paradigm when dealing with rating scales. The former takes the ordering of the categories for granted, while in the latter the order is a hypothesis that needs to be checked (Andrich, 2011). If the categories do not follow the hypothesized order (e.g.…”

“…If the categories do not follow the hypothesized order (e.g. : reversed categories C 1 < C3 < C4 < C2), then they will need to be examined further and improved experimentally (Andrich, 2011). Also, it is necessary to identify whether each category has a probability of being chosen (or marked) greater than the other categories in a specific range of the latent continuum.…”

Detecting meaningful change using the North Star Ambulatory Assessment in Duchenne muscular dystrophy

“…22 This model articulates a theory of how rating scales ought to perform if the values they generate are to be considered scientific measurements. Thus, when the data examined here fit the requirements of the Rasch model (using RUMM 2030 software 23 ), there was evidence that the NSAA was a measurement instrument: all items fit the original construct of ambulation (except 'lifts head' which was removed from the total raw score), targeting was sufficient, all response categories were ordered, the person separation index was high (0.91), there was minimal effect of item dependency on reliability (person separation index 0.89), and there was suitable stability over regime and age.…”