Evaluating the effectiveness of the training program on direct and semi-direct oral proficiency assessment: A case of multifaceted Rasch analysis

Bijani, Houman

doi:10.1080/2331186x.2019.1670592

Cited by 2 publications

(3 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hence, teachers tend not to sustain the standardization element when awarding marks to candidates (Sundqvist et al, 2018). In such a situation, they let themselves use their personal identity and personality even though training has been given (Seker, 2018;Bijani, 2019). That is why literature has also discovered that rater training managed to enhance one's rating consistency but not homogeneity and rater agreement (Eckes, 2015).…”

Section: Discussionmentioning

confidence: 99%

“…Literature has widely reported that one's background can either positively or negatively impact how one rates candidates. Raters' backgrounds include rating experience (Huang et al, 2018;Ahmadi Shirazi, 2019;Şahan and Razı, 2020), training experience (Duijm et al, 2017;Bijani, 2018Bijani, , 2019, teaching experience (Kang and Veitch, 2017;Eckstein and Univer, 2018;Kang et al, 2019), raters' first language (Hijikata-Someya et al, 2015;Marefat and Heydari, 2016;Ahmadi Shirazi, 2019;Kang et al, 2019), familiarity about candidates (Huang et al, 2016;Tanriverdi-Koksal and Ortactepe, 2017;Wikse Barrow et al, 2019), personal traits, gender (Bijani and Khabiri, 2017;Protivínský and Münich, 2018), academic achievement (He et al, 2013;Soltero-González et al, 2016), age (Soltero-González et al, 2016;Isbell, 2017), and cultural background (Stassenko et al, 2014). In terms of raters' experiences, three types of experience are widely examined, which are rating experience, training experience, and teaching experience.…”

Section: Raters' Experiencementioning

confidence: 99%

“…Ideally, with more rating experience, raters' rating quality would be better (Eckes, 2015). Studies on how rating experience can affect raters' ratings have employed many different groups of raters as respondents who are divided into groups based on research contexts (Duijm et al, 2017;Kang and Veitch, 2017;Bijani, 2018Bijani, , 2019Eckstein and Univer, 2018;Huang et al, 2018;Ahmadi Shirazi, 2019;Kang et al, 2019;Şahan and Razı, 2020). Findings from the research have mainly reported that a significant difference was discovered among the rater groups.…”

Section: Raters' Experiencementioning

confidence: 99%

See 2 more Smart Citations

Rater severity differences in English language as a second language speaking assessment based on rating experience, training experience, and teaching experience through many-faceted Rasch measurement analysis

Noh¹,

Matore

2022

Front. Psychol.

View full text Add to dashboard Cite

Evaluating candidates’ answers in speaking skill is difficult and rarely explored. This task is challenging and can bring inconsistency in the rating quality among raters, especially in speaking assessments. Severe raters will bring more harm than good to the results that candidates receive. Many-faceted Rasch measurement (MFRM) was used to explore the differences in teachers’ rating severity based on their rating experience, training experience, and teaching experience. The research uses a quantitative approach and a survey method to enlist 164 English teachers who teach lower secondary school pupils, who were chosen through a multistage clustered sampling procedure. All the facets involving teachers, candidates, items, and domains were calibrated using MFRM. Every teacher scored six candidates’ responses in a speaking test consisting of three question items, and they were evaluated across three domains, namely vocabulary, grammar, and communicative competence. Results highlight that the rating quality was different in terms of teachers’ rating experience and teaching experience. However, training experience did not bring any difference to teachers’ rating quality on speaking test. The evidence from this study suggests that the two main factors of teaching and rating experience must be considered when appointing raters for the speaking test. The quality of training must be improved to produce a rater with good professional judgment. Raters need to be supplied with answer samples with varied levels of candidates’ performance to practice before becoming a good rater. Further research might explore any other rater bias that may impact the psychological well-being of certain groups of students.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Raters' Experiencementioning

confidence: 99%

Section: Raters' Experiencementioning

confidence: 99%

See 1 more Smart Citation

Rater severity differences in English language as a second language speaking assessment based on rating experience, training experience, and teaching experience through many-faceted Rasch measurement analysis

Noh¹,

Matore

2022

Front. Psychol.

View full text Add to dashboard Cite

show abstract

Computer-delivered vs. face-to-face score comparability and test takers’ perceptions: The case of the two English speaking proficiency tests for Vietnamese EFL learners

Nguyen,

Hoang

et al. 2024

Lang Test Asia

View full text Add to dashboard Cite

The present study explored the comparability in performance scores between the computer-delivered and face-to-face modes for the two speaking tests in the Vietnamese Standardized Test of English Proficiency (VSTEP) (the VSTEP.2 and VSTEP.3–5 Speaking tests) according to Vietnam’s Six-Level Foreign Language Proficiency Framework (VNFLPF) and test takers’ experiences. Data were collected from 75 and 82 VSTEP.2 and VSTEP.3–5 university English-majored test takers respectively in both computer-delivered and face-to-face conditions. A counterbalanced research design was adopted to minimise mode order effects. After test completion, 30 of the test takers, 15 from each proficiency test, were interviewed in the focus group format of 3–4 members per group. The results indicated mixed, selective effects of the testing mode. Overall, test scores were comparable in the VSTEP.2 Speaking test but significantly higher in favour of the face-to-face mode for the VSTEP.3–5 Speaking test. However, the statistically significant difference was observed in only one measure of the many analytical criteria (content development in the former test, and pronunciation in the latter test) with mixed mode advantages. The interview data has provided rich refreshing insights into how test takers viewed each testing mode against real-life communication. Their experiences further revealed a wide range of affective preferences involved in the inherent affordances or constraints of each testing mode and their communication and performance/outcome orientation. The findings offer important implications for extrapolation, test preparation and administration, and test taker/rater training in the particular context of the two English speaking proficiency tests in Vietnam and perhaps beyond.

show abstract

Evaluating the effectiveness of the training program on direct and semi-direct oral proficiency assessment: A case of multifaceted Rasch analysis

Cited by 2 publications

References 22 publications

Rater severity differences in English language as a second language speaking assessment based on rating experience, training experience, and teaching experience through many-faceted Rasch measurement analysis

Rater severity differences in English language as a second language speaking assessment based on rating experience, training experience, and teaching experience through many-faceted Rasch measurement analysis

Computer-delivered vs. face-to-face score comparability and test takers’ perceptions: The case of the two English speaking proficiency tests for Vietnamese EFL learners

Contact Info

Product

Resources

About