Evaluation of methods of evaluating test items.

INDICES OF ITEM CONSISTENCY AND VALIDITY1

“…Useful surveys of some twenty indices of validity or consistency are provided by Long and Sandiford (1935) and Guilford (1936~) ; (cf. also Lentz and Hirshstein, 1932 ;Lindquist and Cook, 1 9 3 3~ Zubin, 1934 ;Swineford, 1936a ;Chapanis, 1941). Several of the rather cruder ones, which do not seem to have been applied in any recent researches, will be omitted here.…”

“…Numerous investigations were cariied out in the 1930's into the correlations between different indices applied to the same item-data, and the reliability of tests selected by such indices (Barthelmess, 1931 ;Lentz and Hirshstein, 1932 ;Lindquist and Cook, 1933 ;Henry, Long, 1935 ;Swineford, 1936a, b ;Pintner and Forlano, 1937). Long points out that none of these can be conclusive since the most suitable method varies with the type of test required (e.g., a high or low concentration of middling items), with the level of ability of the population and with the level of difficulty of the items.…”

A Review of Developments and Applications in Item Analysis

“…2.1.2, item difficulty indices have been noted as failing to account for items' measurement heterogeneity (see also Gulliksen 1950, p. 369). Early summaries and lists of item indices (W. W. Cook 1932;Guilford 1936;Lentz et al 1932;Long and Sandiford 1935;Pearson 1909;Richardson 1936;Symonds 1929), and many of the refinements and developments of these item indices from ETS, can be described with little coverage of their implications for test score characteristics. Even when test score implications have been covered in historical discussions, this coverage has usually been limited to experiments about how item difficulties relate to one or two characteristics of test scores (Lentz et al 1932;Richardson 1936) or to "arbitrary indices" (Gulliksen 1950, p. 363) and "arbitrarily defined" laws and propositions (Symonds 1929, p. 482).…”

“…Early summaries and lists of item indices (W. W. Cook 1932;Guilford 1936;Lentz et al 1932;Long and Sandiford 1935;Pearson 1909;Richardson 1936;Symonds 1929), and many of the refinements and developments of these item indices from ETS, can be described with little coverage of their implications for test score characteristics. Even when test score implications have been covered in historical discussions, this coverage has usually been limited to experiments about how item difficulties relate to one or two characteristics of test scores (Lentz et al 1932;Richardson 1936) or to "arbitrary indices" (Gulliksen 1950, p. 363) and "arbitrarily defined" laws and propositions (Symonds 1929, p. 482). In reviewing the sources cited earlier, Gulliksen (1950) commented that "the striking characteristic of nearly all the methods described is that no theory is presented showing the relationship between the validity or reliability of the total test and the method of item analysis suggested" (p. 363).…”

“…Item difficulty and discrimination indices can also be used to check for potential flaws that may warrant item revision prior to item use in test form assembly. The most well-known and utilized difficulty and discrimination indices of item analysis were developed in the early twentieth century (W. W. Cook 1932;Guilford 1936;Horst 1933;Lentz et al 1932;Long and Sandiford 1935;Pearson 1909;Symonds 1929;Thurstone 1925). Accounts of ETS scientists Tucker (1987, p.…”