An Experiment in Science Assessment: Open‐response Versus Fixed‐response Tests

Abstract

The performance of students on two different forms of test of identical subject material was monitored to determine what effect, if any, the form of test had on their performance. Two samples, each of 34 students, consisting respectively of the first year (Sample 1) and second year (Sample 2) of a Polytechnic H.N.C. Chemistry course were tested on two forms of tests, each form covering identical subject material and each test being appropriate for the respective levels of the samples. Each sample was tested firstly with the open-response form of test followed immediately by the equivalent fixedresponse form, all within a period of two hours. The fixedresponse forms of the tests were readministered, without prior warning, after a holiday period of four weeks.The correlation between scores for Parts I and II was 0.88 for the first sample and 0.70 for the second, both significant at 0.1%. Sample 1 scored higher on Part II than Part I though not significantly so (t = 1.76), but Sample 2 scored significantly less on Part II than Part I (t = -2.28, p < .05). For Sample 1 (Test 1) the split-half reliability for the openresponse form (Part I) was 0.89 and for the fixed-response form (Part II) was 0.80. The corresponding figures for Sample 2 (Test 2) were 0.89 for Part I and 0.73 for Part II. Within the fixed-response form of test the multiple-choice items produced significantly higher scores than their open-response equivalents for both samples whereas the reverse was the case for multiple-completion items (p < .01).-57 -Test-retest analysis on Part II (fixed-response) showed no significant difference in mean scores for both samples (p > .05) and significant correlations between test and retest scores (T 1 = 0.75, r 2 = 0.81; p < .001). However, analysis of individual responses showed that on average each student changed over a third of his answers (36%) between test and retest, corresponding to 14 out of 40, with the more difficult items producing the greater number of alterations. Scoring only those items correct on both test and retest resulted in an average drop of 9 marks out of 40, illustrating the weakness of raw scores and the need for standardized scores. Pretested items performed no better than unpretested items on either test or retest, illustrating the sample dependent nature of item statistics.