The reliability of results from national tests, public examinations, and vocational qualifications in England

Abstract

National tests, public examinations, and vocational qualifications in England are used for a variety of purposes, including the certification of individual learners in different subject areas and the accountability of individual professionals and institutions. However, there has been ongoing debate about the reliability and validity of their results. This debate prompted the Office of Qualifications and Examinations Regulation (Ofqual), the regulator of qualifications and examinations in England, to initiate its Reliability of Results Programme in 2008 to investigate the reliability of results from regulated assessments. The purpose of this programme was to gather evidence to develop regulatory policy on assessment reliability in order to improve the national assessment systems in England. This paper provides a brief introduction to the Reliability Programme, discusses its main findings to date and their implications, and explores the impact of the development of regulatory policy on reliability for regulated assessments in England.

Effects of Simulated Interventions to Improve School Entry Academic Skills on Socioeconomic Inequalities in Educational Achievement

“…School entry academic skills in this study assessed both early literacy and maths, meaning that we cannot disentangle the separate effects of literacy and maths on later academic achievement. There is a need for reliability testing of GCSE scoring ( He, 2010 ), although the GCSE outcome measure used in this study is widely recognized in the United Kingdom as being a requirement for further study and higher quality employment opportunities ( Unwin, 2010 ; Wolf, 2011 ).…”

The principal axis approach to value-added calculation

“…Only the dependent variable representing the present achievement is assumed to be a random variable and have an error component involving both sampling error and measurement error. Since both the present performance and the prior performance measures are generally represented by test scores, they will have measurement errors as a result of unreliability of the tests used (see He and Opposs, 2012;He et al, 2013). Both variables should be treated equally as random variables in the analysis (A more sophisticated approach might weigh the line in proportion to the ratio of errors on the two measures.…”

A Nonparametric Approach to Estimate Classification Accuracy and Consistency

“…Two approaches that are popular in providing these estimates are the Livingston & Lewis () (henceforth L&L) approach and the Lee () approach. Examples of using the L&L approach in research and reports for educational assessment systems abound (California Department of Education, , p. 367; Goffreda & DiPerna, ; He & Opposs, ; Irwin, Kingston, Glasnapp, & Poggio, , p. 24; Setzer & He, , p. 14; State of Washington, , p. 139; Wheadon & Stockford, ) as well as for certification assessments (Jaeger, ; World‐Class Instructional Design and Assessment, , p. 443; Wu, Wehrung, & Zumbo, ). The current popularity of the method may, in part, be due to it being a successful first‐mover, as well as its implementation in free (Brennan, ) and proprietary software (e.g., Educational Testing Services' RELCLASS‐COMP).…”