Empirical Evaluation of Computer-Adaptive Alternate Short Forms for the Assessment of Anomia Severity

Abstract

Purpose The purpose of this study was to verify the equivalence of 2 alternate test forms with nonoverlapping content generated by an item response theory (IRT)–based computer-adaptive test (CAT). The Philadelphia Naming Test (PNT; Roach, Schwartz, Martin, Grewal, & Brecher, 1996)was utilized as an item bank in a prospective, independent sample of persons with aphasia. Method Two alternate CAT short forms of the PNT were administered to a sample of 25 persons with aphasia who were at least 6 months postonset and received no treatment for 2 weeks before or during the study. The 1st session included administration of a 30-item PNT-CAT, and the 2nd session, conducted approximately 2 weeks later, included a variable-length PNT-CAT that excluded items administered in the 1st session and terminated when the modeled precision of the ability estimate was equal to or greater than the value obtained in the 1st session. The ability estimates were analyzed in a Bayesian framework. Results The 2 test versions correlated highly ( r = .89) and obtained means and standard deviations that were not credibly different from one another. The correlation and error variance between the 2 test versions were well predicted by the IRT measurement model. Discussion The results suggest that IRT-based CAT alternate forms may be productively used in the assessment of anomia. IRT methods offer advantages for the efficient and sensitive measurement of change over time. Future work should consider the potential impact of differential item functioning due to person factors and intervention-specific effects, as well as expanding the item bank to maximize the clinical utility of the test. Supplemental Material https://doi.org/10.23641/asha.11368040

Computer Adaptive Testing for the Assessment of Anomia Severity

“…We then quantified the agreement between the 30-item PNT-CT and a variable-length CT-generated form of the PNT, called CT-VL, in a follow-up study. 46 The CT-VL was configured to avoid presenting overlapping items from the 30-item PNT-CT administration for a given patient. In addition, for the CT-VL, a stopping rule was set as precision equal to or lower than that obtained by the 30-item PNT-CT.…”

The Aphasia Communication Outcome Measure: Motivation, Development, Validity Evidence, and Interpretation of Change Scores

“…For example, object naming performance by persons with chronic aphasia tends to be very stable over time 58,70,71 and in contrast to these results for the ACOM, IRT-based estimates of measurement error in naming have agreed reasonably well with estimates obtained from test-retest differences. 72,73 It may also be useful in this context to draw a distinction between change scores that exceed measurement error and change scores that exceed those normally found in an untreated sample of individuals. 74 In any case, it is currently recommended that users of the ACOM be cautious in using the SEs provided by the software output when making inferences about change scores for individuals because they may underestimate measurement error due to instability over time.…”

The Utility of Multiplex Closeness Centrality for Predicting Item Difficulty Parameters in Anomia Tests

“…Specifically, Fergadiotis, Hula, Swiderski, Lei, and Kellough ( 2019) demonstrated that scores on the PNT -CAT from 47 PWA were in high agreement with their scores on the full PNT. In addition, Hula et al (2020) developed two equivalent, dynamic, test forms with non-overlapping items for repeated administrations.…”

“…Further, currently available tests invalidly assume constant measurement error based on which 95% confidence intervals are estimated around ability scores. With the exception of recent work (e.g., Hula et al, 2020) on the PNT, naming tests assume equal measurement error regardless of ability level. This ignores that measurement error varies as a function of the degree to which the difficulty of the test targets the ability level of the person being tested (Embretson & Reise, 2000).…”

“…Therefore, a patient may experience frustration or boredom the longer the test is carried out, which may affect their performance and contaminate the test scores. Finally, with limited exceptions (Hula et al, 2020), there is a lack of tools that can generate multiple equivalent test forms with non-overlapping item content. As a result, oftentimes in practice, the same set of testing items are used throughout the course of a treatment, creating a possibility for test practice effects to influence the patients' performance leading to invalid conclusions.…”