Designing computer-based tests: design guidelines from multimedia learning studied with eye tracking

Abstract

The use of computer-based tests (CBTs), for both formative and summative purposes, has greatly increased over the past years. One major advantage of CBTs is the easy integration of multimedia. It is unclear, though, how to design such CBT environments with multimedia. The purpose of the current study was to examine whether guidelines for designing multimedia instruction based on the Cognitive Load Theory (CLT) and Cognitive Theory of Multimedia Learning (CTML) will yield similar effects in CBT. In a within-subject design, thirty-three vocational students completed a computer-based arithmetic exam, in which half of the items were presented in an original design format, and the other half was redesigned based on the CTML principles for instructional design (i.e., adapted format). Results showed that applying CTML principles to a CBT decreased the difficulty of the test items, i.e., students scored significantly higher on the adapted test items. Moreover, eye-tracking data showed that the adapted items required less visual search and increased attention for the question and answer. Finally, cognitive load, measured as silent pauses during a secondary think-aloud task, decreased. Mean fixation duration (a different indicator of cognitive load), however, did not significantly differ between adapted and original items. These results indicate that applying multimedia principles to CBTs can be beneficial. It seems to prevent cognitive overload and helps students to focus on important parts of the test items (e.g., the question), leading to better test results.

Classroom chronicles: through the eyeglasses of teachers at varying experience levels

“…Mental effort can be captured by means of eye tracking as the duration of fixations (Holmqvist et al, 2011). Eye tracking research has proven repeatedly, that shorter durations of fixations indicate a higher mental effort (Gog et al, 2009;Hyönä, 2010;Van Mierlo et al, 2012;Korbach et al, 2016;Dirkx et al, 2021). Indeed, expertise has been shown to be related to longer fixation durations in diverse domains, such as chess, art (Nodine et al, 1993;Reingold et al, 2001;Reingold and Charness, 2005).…”

Aplicación de los principios de la teoría cognitiva del aprendizaje multimedia al diseño de situaciones de aprendizaje y escenarios de formación

“…(Yang et al, 2018;Drees et al, 2020) Comparar el uso de imágenes y audio, frente al uso de imágenes y texto, o la lectura de un guion o artículo (Mirkovski et al, 2019;Lui et al, 2018) Revisar estudios en el área de la salud que utilizan los principios de la TCAM (Mahajan et al, 2020) Validar el efecto de utilizar mapas geográficos digitales interactivos, en lugar del uso de mapas tradicionales (Groshans et al, 2019) Comparar el efecto del aprendizaje por computador (tradicional) frente al aprendizaje por juego digital 3D (Chang et al, 2018) Analizar el efecto de adaptar preguntas de evaluación a partir de los principios de la TCAM para facilitar su comprensión. (Dirkx et al, 2021) Fuente: elaboración propia.…”

IEyeGASE: An Intelligent Eye Gaze-Based Assessment System for Deeper Insights into Learner Performance

“…All these research works focused on learning design and its evaluation but very few works [3,4,28,29,31,34,35,41,42] focused on learner's decision-making, lesson-specific skills and problem-solving skills. The study [2], discussed on use of computer-based tests to support and access student learning. They investigated the effects of adapting computerbased test items according to multimedia learning guidelines.…”

“…Assessment can take different modalities based on the purpose. It can either be summative or formative [1,2]. When an assessment is a part of the teaching process, carried out in a classroom and involves evaluator observation, feedback, homework, etc., it is called a formative assessment.…”