Investigating the validity of oral assessment rater training program: A mixed-methods study of raters’ perceptions and attitudes before and after training

Bijani, Houman

doi:10.1080/2331186x.2018.1460901

Cited by 8 publications

(12 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Almost all of the participants (20 of 21 participants) reported that rater training helped in generating agreement among raters and this is achieved through explanation, discussion, role play, referring to rubric and video samples. This supports Weigle (1999) and Bijani (2018) viewpoint however, Lumley (1998) cautioned that each rater has his or her own interpretation of what a good performance should be which is based on a candidate's abilities. The only one participant (R6) in this study who disagreed was a novice, someone with two years of rating experience and three years of teaching experience, and had this to say "I don't think so because each person has their own reasons on why they mark these students with these marks.…”

Section: Generating Agreementmentioning

confidence: 88%

See 1 more Smart Citation

The impact of rater training on rater reliability in an English oral test

Karuppaiah¹,

Raof²

2020

AJATEL

View full text Add to dashboard Cite

Speaking skill assessment is gaining great interest in the field of assessment nowadays. Literature has highlighted reliability of raters in rating a speaking performance as one of the challenges due to human’s subjective nature. This study has attempted to explore the influence of rater training on rater reliability in the assessment of a spoken task. A qualitative research design was used and, semi-structured interview was employed to obtain data for this study. A total of 21 secondary school teachers participated in the study. They were raters trained to assess an oral English interaction test. Data were analyzed using thematic content analysis which resulted in three main categories i.e. importance of rater training, effects of rater training on rater reliability, and improvement of rater training. The results show that rater training is essential before any rating is to be done, and its effects include, among others, maintaining rating consistency, exposure to test task, and criteria for grading. While suggestions to improve rater training sessions are related to the length, frequency, and quality of training.

show abstract

Section: Generating Agreementmentioning

confidence: 88%

“…One of the main enquiries of a study carried out by Bijani (2018) was investigating the validity of oral assessment rater training programme focusing on raters' perceptions and attitudes. On the whole, it was found that raters' perception of the feedback and training programme received were positive and encouraging.…”

Section: Introductionmentioning

confidence: 99%

The impact of rater training on rater reliability in an English oral test

Karuppaiah¹,

Raof²

2020

AJATEL

View full text Add to dashboard Cite

show abstract

“…Halo effect occurs when raters are inclined to generate equal marks to every domain assessed for a candidate and unable to make fine discrimination among the domains especially when they are tasked to rate using analytical scoring. It is also evident when a particular domain gives influence on another domain either resulting in higher or lower total marks (Bijani, 2018). It is the consequences of raters allowing their global conception of candidates to domineer their judgment and ignore the fact that candidates may be heterogeneous in their mastery of different domains (Hennington, Bradley, Crews, & Hennington, 2013 candidates may be awarded with different marks if they are to be rated using analytical scoring and will challenge the accuracy of raters' rating.…”

Section: Rater Accuracymentioning

confidence: 99%

Brunswik’s Lens Model: This Is How to Inspire Accurate Raters

Noh¹,

Matore²

2019

View full text Add to dashboard Cite

Rating accuracy is one of the fundamental standards in educational assessment to ensure the quality and integrity. Inaccuracy in academic assessment engenders negative implications towards student's motivation and raters' credibility. Therefore, this paper seeks to provide a discussion on rating accuracy in educational assessment based on Brunswik's lens model. The model contends that raters' ratings are not completed directly but through the existence of many factors including raters' variability, rating scales and domains assessed. Raters' idiosyncrasy is scrutinized by describing varied sources that can threaten rating accuracy. This model explains how intervening factors moderate the relationship between candidates' capabilities and observed scores. The discussion may shed some light on the endeavors to inspire raters to be effective and uphold the values of reliable raters through the implementation of thoughtful rater training that incorporates scoring practices, exposure on rater bias and self-directed reflection. Future attempts are necessary for understanding the interaction among intervening factors that influence raters and differences of rating accuracy produced by internal and external raters.

show abstract

“…In other words, rater training was effective on the differential rater severity of DRF. Considering the studies conducted by Bijani (2018), Fahim and and , rater training was effective on DRF. Van Dyke ( 2008) found that the differential rater leniency in the performance assessment process interfered with the measures, but the differential rater severity did not interfere.…”

mentioning

confidence: 96%

“…Başka bir deyişle puanlayıcı eğitimi FPF'in farklılaşan puanlayıcı katılığı üzerinde etkili olmuştur. Bijani (2018), Fahim ve Bijani (2011), ve…”

unclassified

Investigating the Effect of Rater Training on Differential Rater Function in Assessing Academic Writing Skills of Higher Education Students

Şata

Karakaya

2021

Eğitimde Ve Psikolojide Ölçme Ve Değerlendirme Dergisi

View full text Add to dashboard Cite

This study aimed to examine the effect of rater training on the differential rater function (rater error) in the process of assessing the academic writing skills of higher education students. The study was conducted with a pre-test and post-test control group quasi-experimental design. The study group of the research consisted of 45 raters, of whom 22 came from experimental, and 23 came from control groups. The raters were pre-service teachers who did not participate in any rater training before, and it was investigated that they had similar experiences in assessment. The data were collected using an analytical rubric developed by the researchers and an opinion-based writing task prepared by the International English Language Testing System (IELTS). Within the scope of the research, the compositions of 39 students that were written in a foreign language (English) were assessed. Many Facet Rasch Model was used for the analysis of the data, and this analysis was conducted under the Fully Crossed Design. The findings of the study revealed that the given rater training was effective on differential rater function, and suggestions based on these results were presented.

show abstract

Investigating the validity of oral assessment rater training program: A mixed-methods study of raters’ perceptions and attitudes before and after training

Cited by 8 publications

References 24 publications

The impact of rater training on rater reliability in an English oral test

The impact of rater training on rater reliability in an English oral test

Brunswik’s Lens Model: This Is How to Inspire Accurate Raters

Investigating the Effect of Rater Training on Differential Rater Function in Assessing Academic Writing Skills of Higher Education Students

Contact Info

Product

Resources

About