Item Analysis: Theory and Practice Using Classical and Modern Test Theory

Abstract

Scale development is a major part of the work of industrial and organizational psychologists (e.g., developing selection tests or employee attitude surveys), and item analysis is an essential aspect of this process. Item analysis attempts to address "the problem of selecting items for a test, so that the resulting test will have certain specified characteristics" (Gulliksen, 1950, p. 363).In this chapter we demonstrate how to conduct an item analysis using two psychometric frameworks: classical test theory (CTT; Gulliksen, 1950) and item response theory (IRT; Lord and Novick, 1968). Methodology based on IRT (sometimes known as modern test theory or latent trait theory) is relatively new to I-O psychologists. Embretson and Reise have commented on recent changes in psychometrics:Although classical test theory (CTT) has served test development well over several decades, item response theory (IRT) has rapidly become mainstream as the theoretical basis for measurement. Increasingly, standardized tests are developed from IRT due to the more theoretically justifiable measurement principles and the greater potential to solve practical measurement problems. (2000, p. 3) These authors demonstrate how some old rules of measurement based on CTT have been revised or made obsolete by IRT. IRT enables us to solve some problems (e.g., test equating, detecting differential item functioning (DIF), and developing computer adaptive tests) that were difficult or impossible to address with CTT. In terms of item analysis, IRT is more difficult to apply, but offers some advantages over CTT alone. Thus, we advocate that the CTT and IRT approaches be combined in conducting an item analysis. Throughout this chapter, we have tried to present a balanced picture of the advantages and disadvantages, both practical and psychometric, of using CTT and IRT in conducting an item analysis. The remainder of the chapter is divided into the

Short Assessment of Health Literacy—Spanish and English: A Comparable Test of Health Literacy for Spanish and English Speakers

“…IRT is a modern, model-based, and item-oriented psychometric approach to scale development. In addition to testing the psychometric qualities of test items, it has the capability of examining the equivalence of test items between groups, thereby allowing the development of comparable tests (Ellis & Mead, 2002; Embretson & Reise, 2000). …”

“…Second, using an IRT-based approach, test information was computed. Differing from the traditional reliability coefficients (e.g., Cronbach’s alpha), test information reflects how reliably (or precisely) the SAHL-S&E items measure health literacy across the range of literacy (Ellis & Mead, 2002; Embretson & Reise, 2000). …”

Methodology and Validation of Health Literacy Scale Development in Taiwan

“…First, to calibrate the scale, we used item response theory, which is a model-based, item-oriented approach to psychometric test development (Ellis & Mead, 2002;Embretson & Reise, 2000;Zickar, 1998). Item response theory uses information from both the examinees and the item to determine the likelihood that an examinee with a given level of ability (i.e., health literacy) responds correctly to a given item.…”

Development of an Easy‐to‐Use Spanish Health Literacy Test

“…Second, item response theory (IRT) was used to calibrate the SAHLSA. IRT is a model-based and item-oriented approach to psychometric test development (Zickar 1998;Embretson and Reise 2000;Ellis and Mead 2002). It assumes that an examinee's response to an item on a test is related to a latent trait which the test is presumed to measure and that the relationship can be represented by a mathematical function (usually an s-shaped, logistic function) known as an item characteristic curve (ICC).…”