Revisiting the Effect of Varying the Number of Response Alternatives in Clinical Assessment: Evidence From Measuring ADHD Symptoms

Abstract

This study illustrated the effect of varying the number of response alternatives in clinical assessment using a within-participant, repeated-measures approach. Participants reported the presence of current attention-deficit/hyperactivity disorder symptoms using both a binary and a polytomous (4-point) rating scale across two counterbalanced administrations of the Current Symptoms Scale (CSS). Psychometric properties of the CSS were examined using (a) self-reported binary, (b) self-reported 4-point ratings obtained from each administration of the CSS, and (c) artificially dichotomized responses derived from observed 4-point ratings. Under the same ordinal factor analysis model, results indicated that the number of response alternatives affected item parameter estimates, standard errors, goodness of fit indices, individuals’ test scores, and reliability of the test scores. With fewer response alternatives, the precision of the measurement decreased, and the power of using the goodness-of-fit indices to detect model misfit decreased. These findings add to recent research advocating for the inclusion of a large number of response alternatives in the development of clinical assessments and further suggest that researchers should be cautious about reducing the number of response categories in data analysis.

Comparing Irt Properties of Different Likert Scale Lengths: a Case From Attitudinal Measurement on Physics Education Research

“…Methodologically, the performance behavior of different Likert scale lengths can be studied using two approaches either through experimentation (Donnellan and Rakhshani, 2020;Kutscher and Eid, 2020;Shi et al, 2021) or using computer simulation (Goggin and Stoker, 2014;Sangthong, 2020). Experimental study has several advantages that empirically consider the psychological aspects of respondents that were unable to be explained by the simulation study.…”

“…Previous works reported that there were several statistical measures that have been examined to evaluate the experimental setting of different Likert scale lengths. They were reliability (Donnellan and Rakhshani, 2020), discrimination index (Alan and Kabasakal, 2020;Daher et al, 2015;Goggin and Stoker, 2014;Freihat, 2019;Garner, 1960;Weng, 2004), criterion validity (Simms et al, 2019), factor analysis (Shi et al, 2021), inferential statistics (Sangthong, 2020), polytomous item response theory (Kutscher and Eid, 2020), Rasch model (Daher et al, 2015;Ilhan and Guler, 2017), structural equation modeling (Maydeu-Olivares et al, 2009), and information theory (Green and Rao, 1970;Miller, 1956;Parducci and Wedell, 1986) (Hambleton et al, 1991;Harwell and Gatti, 2001;Lee and Paek, 2014).…”

Are the Steps on Likert Scales Equidistant? Responses on Visual Analog Scales Allow Estimating Their Distances

“…Such response levels are regarded as ordinal steps along a presumed continuous dimension and the Likert format is generally preferred over a simpler yes–no format that forces respondents to express only all–or–none positions. Thus, switching from just two (i.e., yes–no) to more response alternatives per item is regarded as allowing for finer granularity that, in turn, might result in more precise measures or more ability to discriminate respondents from one another (see, for example, Alan & Kabasakal, 2020 ; Donnellan & Rakhshani, 2023 ; Maydeu-Olivares et al, 2009 ; Müssig et al, 2022 ; Shi et al, 2021 ). In the limit, a continuous response format can be implemented via a visual analog scale (VAS) on which respondents mark any position along a continuous line, limited only by the resolution with which the mark can be made and subsequently read off.…”