Maintaining and monitoring quality of a continuously administered digital assessment

Abstract

Digital-first assessments are a new generation of high-stakes assessments that can be taken anytime and anywhere in the world. The flexibility, complexity, and high-stakes nature of these assessments pose quality assurance challenges and require continuous data monitoring and the ability to promptly identify, interpret, and correct anomalous results. In this manuscript, we illustrate the development of a quality assurance system for anomaly detection for a new high-stakes digital-first assessment, for which the population of test takers is still in flux. Various control charts and models are applied to detect and flag any abnormal changes in the assessment statistics, which are then reviewed by experts. The procedure of determining the causes of a score anomaly is demonstrated with a real-world example. Several categories of statistics, including scores, test taker profiles, repeaters, item analysis and item exposure, are monitored to provide context and evidence for evaluating the score anomaly as well as assure the quality of the assessment. The monitoring results and alerts are programmed to be automatically updated and delivered via an interactive dashboard every day.

Duolingo English Test :Technical Manual

“…The DET utilizes a custom-built psychometric quality assurance system, Analytics for Quality Assurance in Assessment (AQuAA; Liao et al, 2022), to continuously monitor test metrics and trends in the test data. AQuAA is an interactive dashboard that blends educational data mining techniques and psychometric theory, allowing the DET's psychometricians and assessment scientists to continuously monitor and evaluate the interaction between the test items, the test administration and scoring algorithms, and the samples of test takers, ensuring scores are consistent over many test administrations.…”

Quality management of higher education within the framework of the socio-investment model of economic growth: State audit and financial control

“…The socio-investment model of economic growth is manifested in the following: the development of digital competencies among the population, the training of digital personnel for business, lifelong learning, targeted, and corporate (requested and paid by employers) training to improve the qualification level of employees of enterprises. All this is implemented based on universities (Henze et al, 2022;Liao et al, 2022).…”