In practices of direct assessment of writing ability, the variability of human decision-making during scoring poses great challenges to the validity of assessment (Kane, 2006). The variables causing differences in individual raters' scoring interpretations have been widely investigated (e.g. Eckes, 2012; Wolfe et.al, 2016). However, the issue of how raters negotiate to resolve discrepancies has not received attention although rater negotiation is a widely used score resolution method. As it has been emphasized by scholars interested in the argumentative behavior of raters (e.g. Trace et. al., 2017), a systematic analysis of score negotiations will enable us to analyze the dependability of score negotiations. The purpose of this study is twofold: to present a thorough analysis of the argumentative structure of rater discrepancy resolution discussions with a view to understanding their underlying dynamics, and to investigate whether the elements of the argumentative structure of negotiations differ from research settings to authentic score resolution practices. In line with this aim, rater negotiations following a written test at the language school of an English-medium university were analyzed within the framework of Argumentation Theory by Toulmin (1958) and Walton (2005Walton ( , 2016. The negotiation data were obtained from 99 recorded rater discussions among 30 EFL teachers, and transcribed, coded and categorized into argumentative discussion moves. A Rater Negotiation Scheme (RNS) was developed through a recursive data analysis and categorization process, and it was validated through field-testing in authentic settings. The findings have implications both for research on rater negotiations and arguments on the reliability of the method.
This chapter reports on participant evaluations of an online professional development course on language assessment literacy (LAL) for practitioners interested in the field. With the introduction and widespread of online courses following the Covid era, the need to evaluate the effectiveness of the new modalities and platforms to deliver these courses as well as the online course content has become inevitable. To serve for this purpose, this chapter presents the analysis of data collected from a multinational online community of language teachers through the end-of-course questionnaire (n=13) and focused interviews (n=6). The results of the analysis suggested three main issues indicated by the course participants: learning in an online community, course design, and course content. The chapter which discusses the emerging sub-themes has implications for teacher educators, researchers, and course designers interested in learning about the participant perspective towards an online course in a multicultural LAL context.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.