The state‐of‐the‐art in digital technology‐based assessment

Abstract

The role of digital technology in assessment has received a great deal of attention in recent years. Naturally, technology offers many practical benefits, such as increased efficiency with regard to the design, implementation and scoring of existing assessments. More importantly, it also has the potential to have profound, transformative effects on the field of assessment by facilitating the integration of formative activities with accountability requirements and broadening the range of abilities and the scope of constructs that can be assessed. This article provides an overview of the current state-ofthe-art in digital technology-based assessment, with particular reference to advances in the automated scoring of constructed responses, the assessment of complex 21 st century skills in large-scale assessments and innovations involving high fidelity virtual reality simulations. Key challenges with respect to each are highlighted before the extent to which digital technology is truly transforming assessment is considered. | I NTR OD U CTI ONThe interconnectedness of technology and assessment 1 has a long history. Madaus (2001) refers to a technology as any body of special knowledge, skills and procedures that people use to satisfy a need, solve a problem or attain a societal, economic, or educational goal. Under this definition, it is clear that educational assessment itself is a technology and, as Kellaghan and Madaus (2003) point out, one that has existed since an external civil service examination system was invented in China around 1100 BC. While the system changed over the centuries until its demise in 1905, the general approach involved candidates writing up to eight essays within an allocated time period in which they explained ideas from the Four Books and Five Classics of Confucianism (www.sacu.org/examinations.html). In many respects, the artefacts and processes of assessment that make it a technology have been visible from the very beginning in the form of test booklets and answer sheets, as well as in the specialist knowledge, skills and language of its community of practitioners. What has changed in the recent past is the speed with which one technology (digital technology 2 ) is transforming another (assessment) in terms of the constructs that can be measured and the types of 160 | How to cite this article: O'Leary M, Scully D, Karakolidis A, Pitsia V. The state-of-the-art in digital technologybased assessment. Eur J Educ. 2018;53:160-175. https://doi.

Animated videos in assessment: comparing validity evidence from and test-takers’ reactions to an animated and a text-based situational judgment test

“…The technological developments of the last decades have significantly influenced assessment, improving test development, administration, and scoring (Norcini et al, 2013;O'Leary et al, 2018). Through the use of technology, assessments can increase both the quantity and quality of the 1 SJTs are assessments that provide test-takers with descriptions of challenging real-life situations (in the form of vignettes), followed by a number of possible alternative ways to deal with them (Motowidlo et al, 1990).…”

Terminological “Communities”: A Conceptual Mapping of Scholarship Identified With Education’s “Global Turn”

“…The community that includes these topics demonstrates that the OECD and the Program for International Student Assessment (PISA), which have begun assessing 21st-century skills and has taken a wide interest in their dissemination, are most commonly mentioned in tandem with topics related to technology and digital skills—even more often than they are mentioned alongside topics like assessment or any nontechnological skills. This may indicate a certain inclination of these organizations to promote a particular type of 21st-century student (Lucas, 2016; O’Leary et al, 2018). This tendency raises some questions about inequality between nations and within them in terms of the digital divide, a research concern that also appears in this community, and highlights the essential role that access to information plays in promoting 21st-century skills (Scalise, 2016; Storksdieck, 2016; Terrazas-Arellanes et al, 2017).…”

“…Many articles in this cohort discuss technological proficiency and information literacy as key skills required of 21st-century learners (e.g., Byker & Marquardt, 2016; Ruggiero & Mong, 2015; Scalise, 2016). Others discuss technology integration in the assessment of 21st-century skills, comprising another strand of research that emerged from the network analysis (e.g., O’Leary et al, 2018).…”

Automated scoring of junior and senior high essays using Coh-Metrix features: Implications for large-scale language testing

“…Feature extraction is the first and most important step which dictates the efficiency of any AES system. Recent studies have highlighted the fact that state-of-the-art, operational AES programs use different procedures for extracting and modeling the features of essay language (Bunch, Vaughn, & Miel, 2016; Chen, Fife, Bejar, & Rupp, 2016; Foltz, 2016; Latifi, 2016; O’Leary, Scully, Karakolidis, & Pitsia, 2018). Despite the use of different methods for feature extraction, Shermis (2014a) concluded that they produce comparable results for scoring student-produced essays.…”

“…Surface-level features (such as word frequencies and length) are in use by some AES systems (e.g., O’Leary et al., 2018), but they are often challenged by the educational community because their rationale and empirical relation with human scores and with other traits of writing quality are not well defined (Attali, 2013; Cohen, Levi, & Ben-Simon, 2018; Perelman, 2014). Recent advances in computational linguistics and natural language processing (NLP) have given rise to more rational methods for extracting features that go beyond surface-level characteristics.…”