No abstract

There is a growing body of research literature that considers how the mode of assessment, either computer-based or paper-based, might affect candidates' performances. Despite this, there is a fairly narrow literature that shifts the focus of attention to those making assessment judgements and which considers issues of assessor consistency when dealing with extended textual answers in different modes. This research project explored whether the mode in which a set of extended essay texts were accessed and read systematically influenced the assessment judgements made about them. During the project, 12 experienced English literature assessors marked two matched samples of 90 essay exam scripts on screen and on paper. A variety of statistical methods were used to compare the reliability of the essay marks given by the assessors across modes. It was found that mode did not present a systematic influence on marking reliability. The analyses also compared examiners' marks with a gold standard mark for each essay and found no shifts in the location of the standard of recognised attainment across modes. IntroductionLiterature suggests that the feasibility, validity and reliability of working on screen have long been the focus of audiences from a wide range of backgrounds, ranging from studies in the contexts of education to those in occupational and cognitive psychology, among others. This particular study sought to investigate whether the mode of marking (on screen or paper) had any influence on essay marking reliability and markers' leniency/rigour.

This study investigated the functions of annotations, the role of annotations in markers' decision‐making processes, whether annotations conform to conventions, and whether these vary according to subject area. Across subjects a number of scripts were analysed to survey which annotations are subject specific and which are more general. Twelve examiners at different levels of responsibility marked the same General Certificate of Secondary Education Mathematics and Business Studies scripts as one another. The marking was analysed to identify whether there were any patterns in the use of annotations. Retrospective individual interviews were used to inform the analysis of the annotations. Findings showed that the frequency and type of annotations used differed between subjects and that within a subject team examiners drew from a common pool of annotations. Purposes for annotation use included supporting markers' judgements and explaining decisions.

Within large scale educational assessment agencies in the UK, there has been a shift towards assessors marking digitally scanned copies rather than the original paper scripts that were traditionally used. This project uses extended essay examination scripts to consider whether the mode in which an essay is read potentially influences the judgements made about it, employing qualitative and quantitative data-gathering approaches to focus on the annotation practices of readers who are assessing extended essays, and to explore whether this might be affected by mode. The project also gathers evidence about the spatial encoding ability of these readers to suggest that mode-related influences on annotation might influence the ability of readers to comprehend fully the material being read. Examiners had less spatial awareness about the location of script features on screen than on paper; however, annotating on screen might link to both the provision of appropriate annotations and the relative ease with which these are deployed whilst reading. There was also evidence that on-screen annotation could contribute to the development of good mental representations of texts on screen. IntroductionThere is a growing body of research literature that considers how the mode of assessment, either computer-or paper-based, might affect students' performances (Paek 2005). Despite this, there is a fairly narrow literature that shifts the focus of attention to those making assessment judgements and which considers issues of assessor behaviour when dealing with extended written essay answers in different modes. This might be considered surprising since research literature from domains such as ergonomics, human factors, human-computer interaction, and the psychology of reading suggests that the mode in which longer texts are read might be expected to influence the way that readers access and comprehend such texts (324 M. Johnson and R. Nádas ability of these readers to suggest that mode-related influences on annotation might influence the ability of readers to fully comprehend the material being read. This issue has particular relevance for the domain of educational assessment where there are moves towards increased digital script marking, as annotation has a clear formal quality assurance function within large scale assessment systems. What is less clearly articulated is the informal utility of annotations for assessors whilst comprehension building, a function that carries implications beyond assessment and into the realms of human cognition, cognitive load and the psychology of reading. BackgroundBennett (2002) describes the rapid growth of computer technology use in workplaces and education as inexorable. Although technology offers the potential to broaden educational assessment beyond what traditional methods allow, there are inevitable concerns during a transition phase (where assessments exist in both paper-and computer-based modes) that their outcomes are not comparable. In her review of comparability studies since 1993, Paek (2005 ...

No abstract