Classification Consistency and Accuracy for Complex Assessments Using Item Response Theory

Lee, Won‐Chan

doi:10.1111/j.1745-3984.2009.00096.x

Cited by 58 publications

(151 citation statements)

References 29 publications

Supporting

Mentioning

141

Contrasting

Unclassified

Order By: Relevance

“…It should be realised that it is the performance on the overall qualification that is used to make inferences about the level of attainment in a subject for the test takers. Table 6 shows the classification accuracy and consistency measures estimated by Wheadon and Stockfort (2011) for 20 GCE components offered by the Assessment and Qualifications Alliance (AQA) in 2008 and 2009, again using the methods employed by Livingston and Lewis (1995) and Lee (2010), which show considerable variation between the different subjects studied. As expected, the classification consistencies for the components are substantially lower than the classification accuracies.…”

Section: Evidence Of Reliability In Gcse and Gcementioning

confidence: 99%

The reliability of results from national tests, public examinations, and vocational qualifications in England

He¹,

Opposs²

2012

Educational Research and Evaluation

View full text Add to dashboard Cite

National tests, public examinations, and vocational qualifications in England are used for a variety of purposes, including the certification of individual learners in different subject areas and the accountability of individual professionals and institutions. However, there has been ongoing debate about the reliability and validity of their results. This debate prompted the Office of Qualifications and Examinations Regulation (Ofqual), the regulator of qualifications and examinations in England, to initiate its Reliability of Results Programme in 2008 to investigate the reliability of results from regulated assessments. The purpose of this programme was to gather evidence to develop regulatory policy on assessment reliability in order to improve the national assessment systems in England. This paper provides a brief introduction to the Reliability Programme, discusses its main findings to date and their implications, and explores the impact of the development of regulatory policy on reliability for regulated assessments in England.

show abstract

Section: Evidence Of Reliability In Gcse and Gcementioning

confidence: 99%

The reliability of results from national tests, public examinations, and vocational qualifications in England

He¹,

Opposs²

2012

Educational Research and Evaluation

View full text Add to dashboard Cite

show abstract

“…Lathrop and Cheng (2013) summarized and compared two major lines of such research: one focuses on decisions made on the basis of number-correct scores (Lee, 2010;Lee, Hanson, & Brennan, 2002), and the other on the basis of latent trait estimate, i.e.,θ (Runder, 2001(Runder, , 2005Guo, 2006;Martineau, 2007;Wyse & Hao, 2012). Lathrop and Cheng (2013) showed with simulation studies that in most conditions the two approaches yield similar results.…”

Section: The Irt Modelsmentioning

confidence: 99%

Standard Error of Ability Estimates and the Classification Accuracy and Consistency of Binary Decisions

2014

View full text Add to dashboard Cite

While estimation bias is a primary concern in psychological and educational measurement, the standard error is of equal importance in linking key aspects of the assessment structure, especially when the assessment goal concerns the classification of individuals into categories (e.g., master/non-mastery). In this paper, we show analytically how standard error of ability estimates affects expected classification accuracy and consistency when the decision is binary. When standard error decreases, the conditional classification accuracy and consistency increase. Given an examinee population and a cut score, smaller standard error over the entire latent trait continuum guarantees higher overall expected classification accuracy and consistency. We were also able to show the interrelationship between standard error, the expected classification consistency, and reliability. Utilizing the relationship between standard error and expected classification accuracy and consistency, we derive the upper bounds of the overall expected classification accuracy and consistency of a fixed-length computerized adaptive test. The lower bound of the expected classification accuracy and consistency is also derived given a number of stopping rules of variable-length computerized adaptive testing. Implications of these analytical results on operational tests are discussed.

show abstract

“…It must be noted that classification accuracy is not the same as classification consistency. Where classification accuracy compares the true classification of a student with the observed classification, classification consistency compares the classification in two (parallel) exams (see also Lee, 2008). Here, the interest is solely in classification accuracy.…”

Section: Introductionmentioning

confidence: 99%

“…Furthermore, it is more difficult to correctly categorize students close to the standard than students far away from the standard (Martineau, 2007). Therefore, a graph of classification accuracy should show a dip around where the standard meets the average ability in the population (Lee, 2008).…”

Section: Introductionmentioning

confidence: 99%

Towards an Integrative Formative Approach of Data-Driven Decision Making, Assessment for Learning, and Diagnostic Testing

Hubregtse¹,

Eggen²

2012

Psychometrics in Practice at RCEC

View full text Add to dashboard Cite

Classification accuracy of single exams is well studied in the educational measurement literature. However, when making important decisions, such as certification decisions, one usually uses several exams: an exam set. This chapter elaborates on classification accuracy of exam sets. This is influenced by the shape of the ability distribution, the height of the standards, and the possibility for compensation. This is studied using an example from vocational education and training (VET). The classification accuracy for an exam set is computed using item response theory (IRT) simulation. Classification accuracy is high when all exams from an exam set have equal and standardized ability distributions. Furthermore, exams where few or no students pass or fail increase classification accuracy. Finally, allowing compensation increases classification accuracy.

show abstract

Classification Consistency and Accuracy for Complex Assessments Using Item Response Theory

Cited by 58 publications

References 29 publications

The reliability of results from national tests, public examinations, and vocational qualifications in England

The reliability of results from national tests, public examinations, and vocational qualifications in England

Standard Error of Ability Estimates and the Classification Accuracy and Consistency of Binary Decisions

Towards an Integrative Formative Approach of Data-Driven Decision Making, Assessment for Learning, and Diagnostic Testing

Contact Info

Product

Resources

About