The Effect of Differential Weighting of Individual Item Responses on the Predictive Validity and Reliability of an Aptitude Test

Abstract

An empirical investigation of the effect of choice weight scoring on predictive validity and reliability. Choice weight scoring refers to the procedure whereby different weights may be assigned to all the options of an item. Four groups of subjects were included in the experiment. Weights derived from each group were used to score tests for another group in order to assess the cross-validity of the weighted scoring. In no case did the increments in reliability and validity due to the weighted scoring exceed .03,

Partial-Credit Scoring Methods for Multiple-Choice Tests

“…Echternacht (1976) did not find validity weakened by empirical option weighting but used a criterion so similar to the predictor test that the reported validity coefficients might better be regarded as parallel-form reliability coefficients. Another exception (not covered by was reported by Sabers and White (1969), who obtained nearly identical reliability and validity estimates for number-right versus option-weighted scores. This study and the Davis and Fifer (1959) study differed from the other studies involving validity in that no admonition against random guessing was given.…”

Empirical Choice Weighting Under "Guess" and "Do Not Guess" Directions

“…Empirical choice-weighting resulted in substantial improvement in reliability over conventional scoring except in the study by Sabers and White 1969 which showed no appreciable difference. With respect to validity, the results are not as consistent nor as promising.…”

“…There have been a number of more recent studies which have investigated empirical choice-weighting without judgmental modification of the resulting weights (Bayuk, 1974;Echternacht, 1976;Hendrickson, 1971;Reilly and Jackson, 1973;Sabers and White, 1969). Empirical choice-weighting resulted in substantial improvement in reliability over conventional scoring except in the study by Sabers and White 1969 which showed no appreciable difference.…”

“…Since empirical choice-weighting tends to increase the homogeneity of the test scores, it appears counter-productive to use such scoring to improve the correlation between scores from an aptitude test and scores on a complex criterion such as grade-point-average. The Davis and Fifer (1959) study and the study by Sabers and White (1969) differ in another potentially important respect from the other studies; namely, the others used tests with directions which advise against completely random guessing and ordinarily use CFG scores. Consequently, in these studies, it was necessary to compute weights not only for the printed choices but also for the choice omit.…”

“…Davis and Fifer (1959) used teacher ratings of arithmetic ability and a free-answer form of the predictor test to estimate validity. Sabers and White (1969) used end-of-semester grades in an algebra course to determine the predictive validity coefficient for the Iowa Algebra Aptitude Test. These are the only two studies that used criteria obtained independently of the predictor other than college grade-pointaverage.…”