Investigating Factors Affecting the Uptake of Automated Assessment Technology

Abstract

Executive SummaryAutomated assessment is an emerging innovation in educational praxis, however its pedagogical potential is not fully utilised in Australia, particularly regarding automated essay grading. The rationale for this research is that the usage of automated assessment currently lags behind the capacity that the technology provides, thus restricting the pedagogical benefits for students, job satisfaction for staff, and quality assurance and financial benefits for universities. The exploration of the different perspectives of stakeholders regarding their needs and expectations of automated assessment shows the high-stake game of surviving and thriving. Inter alia, students value education as a means to gain employment, educators' value education as their means of employment, and universities value education as their means of existence. The various facets of this interplay are described, including the value of knowledge and learning (pedagogical considerations); society's system for measuring worth and exchanging value (economic considerations); sociotechnological evolution and emerging innovations in educational assessment, and the adoption and resistance thereof.We contribute to the field a national survey that investigated and reveals potential reasons for this sub-optimal utilisation by exploring the candid perceptions of Australian university students and teachers regarding automated assessment. Students and staff from Australian universities were invited to participate in an anonymous online survey. Two hundred and sixty five people completed the survey. The analysis utilised a mixedmethod approach, combining quantitative and qualitative research paradigms. We present a sub-set of the survey results; this sample's use of automated

The role of attitudinal factors in mathematical on-line assessments: a study of undergraduate STEM students

“…This is combined in a factor called 'received feedback' (RF) which stands for "the student's belief that the support and feedback they receive from the on-line platform will enhance their learning". (Dreher et al, 2011) remark on the pedagogical benefits obtained, particularly surrounding feedback practices, by using e-assessment technologies. Technological tools allow students can in turn be freed to determine their own learning path along defined milestones and support them by assessing their learning for successful performance.…”

“…This issue combined with the diversity of students' academic backgrounds supports the need for formative electronic assessment (Miller, 2009). Immediate on-line feedback is an essential feature of such assessments: It has positive effects on students' learning performance; it activates their intrinsic motivation (Dreher, Reiners & Dreher, 2011) and helps them to achieve their goals (Whitelock & Brasher, 2006). Since effective use of on-line feedback is clearly desirable for all stakeholders, it is important to explore the students' perception of the on-line feedback.…”

Ethical Implementation of an Automated Essay Scoring (AES) System: A Case Study of Student and Instructor Use, Satisfaction, and Perceptions of AES In a Business Law Course

“…Despite the growth and improvement in AES technology acceptance in academia has been slow. Dreher, Reiners and Dreher (2011) found that while most instructors can define an AES, most have never used one for scoring essays and have no interest in using one. The vast majority of instructors who have used an AES have done so for correcting multiple choice exams through the use of optical character scanning recognition (Dreher, Reiners & Dreher, 2011).…”

“…Assessment of student learning is an integral part of teaching and also the most time consuming. As Dreher, Reiners and Dreher (2011) state, "assessment guides the teaching and learning process by providing reciprocal feedback to both educators and students so that they may improve in their respective tasks" (p. 162). Assessment can serve two purposes either formative or summative.…”

“…Multiple choice questions can assess knowledge and understanding, but the higher levels of application, analysis, synthesis and evaluation require essay and discussion questions. These higher level functions require transfer of theory to practical situations (application), identification of relevant components and logic in the learning material (analysis), combining information to produce new products (synthesis) and making decisions that create an impact on a given application (evaluation) (Dreher, Reiners & Dreher, 2011). Student success is increased when they are "given challenging, real-world practice assignments with rapid, meaningful feedback" (Matthews, Janicki & Patterson, 2012, p. 73).…”