Comparing paper‐based and electronic multiple‐choice examinations with personal devices: Impact on students' performance, self‐efficacy and satisfaction

Abstract

Nowadays the size of university classes along with the growing number of students to be tested for final examination raise unprecedented challenges relating to the management, monitoring and evaluation of learning. Technology may provide some solutions that deserve to be investigated. This paper explores the potentialities and limitations of Computer-Based Testing (CBT), specifically BYOD e-test, compared to traditional Paper-Based Testing (PBT) to verify whether, and to what extent, an electronic mode of assessment can become a suitable alternative to PBT. It is based on a study carried out at the University of Florence during 2016/17. 606 students participated in the study, of whom 443 opted for CBT using their own devices, while 163 preferred PBT. Participants who experienced CBT also answered a survey on perceptions, self-efficacy and satisfaction. The results show that students' performances were better with CBT, and that a positive relationship exists between the perceived level of self-efficacy and the propensity to adopt digital tests. In addition, students greatly appreciated the electronic system, especially for the possibility of immediate feedback. Some critical issues emerged relating to on-screen reading, which suggests the need for careful design of testing tools.

Tablet assessment in primary education: Are there performance differences between TIMSS' paper‐and‐pencil test and tablet test among Dutch grade‐four students?

“…Although this study did not find a mode effect, it does not mean that some of the advantages and disadvantages of CBT described in the literature did not occur during the tablet assessment. A study of university students revealed that although students preferred the CBT and generally performed better on it, they also experienced limitations, mainly because the test items were presented on a screen (Nardi & Ranieri, 2018). During the data collection for the present study, some of the advantages and disadvantages of the tablet assessment were reported by the Dutch test administrators.…”

“…The study also examined the occurrence of a mode effect between mathematics items with a high, medium or low reading demand (Mullis et al, 2013;Punter et al, 2018). The literature review suggested that the complexity and length of reading texts on a computer or tablet screen were two of the main disadvantages of CBT and could be related to the lower performance of students on CBTs compared to PBTs (Choi & Tinkler, 2002;Kingston, 2009;Ling, 2016;Mangen et al, 2013;Nardi & Ranieri, 2018). Therefore, a negative mode effect for the tablet test was expected for mathematics items classified as having a high reading demand.…”

“…This “mode effect” refers to the likelihood of differential student performance due to differences in how items are presented in PBT versus CBT (Wang, Jiao, Young, Brooks, & Olson, 2007). The results of these studies were mixed, as differences in student achievement were often found to favour PBT, with some studies reporting higher achievement in CBT or no performance differences at all (Bennett et al , 2008; Clariana & Wallace, 2002; Jerrim, 2016; Nardi & Ranieri, 2018; Noyes & Garland, 2008; Russell, Goldberg, & O'Connor, 2003; Wang et al , 2007). Research suggests that the occurrence of mode effects depends on the characteristics of the students taking the test (eg, gender, socio‐economic status or experience with the device) as well as the test characteristics, such as the subject matter being tested, the user‐friendliness of the testing device, and how the test items are presented.…”

“…Mullis, Martin, and Foy (2013) found a wide achievement gap between low reading‐demand items and high reading‐demand items in the TIMSS 2011 PBT in mathematics. This effect may even be stronger for digital assessments in which students read from a screen (visual fatigue) and often have to scroll (Kingston, 2009; Nardi & Ranieri, 2018). Reading on a tablet computer can take even longer than on a desktop or laptop because the screen is smaller and, therefore, requires more scrolling than with a desktop computer (Ling, 2016).…”

E-Learning, MALL and LMOOC or a New Didactic Tendency in Foreign Language Learning: A Meta-Analysis from 2014 to 2018

“…Therefore, e-learning activities must be integrated into traditional learning; otherwise they will not be successful, if based on the spontaneity of the study. Studies of FLL are more focused on the research of European foreign languages and the growing interest and enthusiasm of teachers and students is considered a rich sector to be explored in future research also for its intercultural nature (Nardi & Ranieri, 2019).…”