Development and validation of a rating scale for Iranian EFL academic writing assessment: a mixed-methods study

Abstract

The present study reports the process of development and validation of a rating scale in the Iranian EFL academic writing assessment context. To achieve this goal, the study was conducted in three distinct phases. Early in the study, the researcher interviewed a number of raters in different universities. Next, a questionnaire was developed based on the results of the interview along with the related literature. Later, the questionnaire was sent to thirty experienced raters from ten major state universities in Iran. Results of the country-wide survey in this phase showed that there was no objective scale in use by the raters in the context. Therefore, in the second development phase of the study, fifteen of the raters who participated in the first phase were asked to verbalize their thoughts when each rating five essays. At the end of this phase, a first draft of the scale was developed. Finally, in the last validation phase of the study, ten raters were asked to each rate a body of twenty essays using the newly developed scale. Next, eight of the raters participated in a follow-up retrospective interview. The analysis of the raters’ performance using FACETS showed high profile of reliability and validity for the new scale. In addition, while the qualitative findings of the interviews counted some problems with the structure of the scale, on the whole, the findings showed that the introduction of the scale was well-received by the raters. The pedagogical implications of the study will be discussed. In addition, the study calls for further validation of the scale in the context.

Iranian EFL teachers’ perceptions of writing assessment literacy: a countrywide study

“…According to them, in order to enhance Iranian EFL teachers' competence and preparation for writing assessment, there is a great need for significant improvement in EFL teacher education programs and recruitment. As an example of poor writing assessment literacy among Iranian EFL writing teachers, Ghanbari and Barati (2020) found that there was no objective rating scale in use among the raters in the Iranian EFL academic writing assessment. Based on this, they suggested that higher educational planning bodies in the context should consider the design and implementation of rater training courses.…”

“…This was mostly due to the missing place of a special training program for writing teachers. Similarly, studies in different contexts (Alkharusi et al, 2012;Ghanbari & Barati, 2020;Jannati, 2015;Tayyebi & Moradi Abbasabadi, 2020;Taylor, 2009;Valizadeh, 2019) have revealed that many writing teachers receive little or no formal assessment training in the preparatory programs and often they do not have enough knowledge about assessment. Similar to the findings of the present study, the literature shows that in different EFL/ESL writing assessment contexts, the pre-service and in-service teachers are not provided with a holistic and experiential training program that enables them to integrate the theoretical knowledge of the teachers with the realities of their practice.…”

“…Moreover, the traditional transmission-based approach of teacher education programs hinders the formation of a dynamic practical knowledge base among teachers. Lack of a formal assessment training component is evident in the Iranian EFL assessment context (Alkharusi et al, 2012;Ghanbari & Barati, 2020;Jannati, 2015;Volante & Fazio, 2007). For instance, Iranian teachers believe that in-service programs help them little and pointed out to the poor quality of the training system (Atai & Mazlum, 2013;Baniasad-Azad et al, 2016;Nezakat-Alhossaini and Ketabi, 2013).…”

“…In fact, present in-service EFL teacher education programs and even EFL university programs in Iran are not effective in improving English teachers' assessment skills. Therefore, such courses and programs in Iran certainly need revision or reformation (Aliakbari & Ghoreyshi, 2013;Ghaemi et al, 2016;Ghanbari & Barati, 2020;Moiinvaziri & Razmjoo, 2016;Razi & Kargar, 2014;Soltanpor & Valizadeh, 2019). Thus far, little attention has been paid to writing assessment preparation in ESL/EFL contexts (Lee, 2017).…”

Raters’ perceptions of rating scales criteria and its effect on the process and outcome of their rating

“…The outcome of the writing assessment derives from an interaction among many elements of assessment including rater, test taker, scale, scoring, and practice. Therefore, the quality of rating has a significant bearing for the interpretation of outcomes (Ghanbari & Barati, 2020 ; Hamp-Lyons, 2007 ), and hence, different interactions regarding all these aspects of assessment can lead to different outcomes. In writing assessment, the impact of rating scale on the rating process (Barkaoui, 2010b ), complexity of rating criteria (Joe et al, 2011 ), and raters’ perception can induce the lowest inter-rater reliability.…”