The Impact of Rater's Personality Traits on Holistic and Analytic Scores: Does Genre Make any Difference too?

Alaei, Mahnaz Mostafaei; Ahmadi, Masoume; Zadeh, Naser Sabourian

doi:10.1016/j.sbspro.2014.03.539

Cited by 4 publications

(6 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, it should be noted that those two studies recruited peer-reviewers of undergraduate students instead of using native or expert raters, which is a more conventional practice in L2 writing assessment contexts. Moreover, previous studies have used a sample size similar to the present study, such as Carrell (1995) ( n = 20) and Alaei et al (2014) ( n = 31).…”

Section: Discussionmentioning

confidence: 99%

“…It is notable that many studies (e.g., Bernardin et al, 2000, 2009, Alaei et al, 2014) applying the trait approach to personality preferred the NEO-FFI (i.e., a 60-item shortened version of NEO-PI-R) based on practical considerations. While offering convenience, NEO-FFI does compromise its reliability and validity (Aluja et al, 2005).…”

Section: Literature Reviewmentioning

confidence: 99%

“…A milestone on this path was established by Kane and colleagues (1995), when a particular personality trait of raters (i.e., agreeableness) was suspected and proved for the first time to be an important cause of rater leniency. Strangely enough, for decades after 1995, only very few related studies have been seen in publications (i.e., Alaei et al, 2014; Bernardin et al, 2000, 2009; Carrell, 1995; Yun et al, 2005). Moreover, findings in these studies are inconsistent or even conflicting with each other.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Methodologically Improved Study on Raters’ Personality and Rating Severity in Writing Assessment

2021

View full text Add to dashboard Cite

Personality is an inherent rater’s characteristic influencing rating severity, but very few studies examined their relationship and the findings were inconclusive. This study aimed to re-investigate the relationship between raters’ personality and rating severity with more control on relevant variables and more reliable analysis of rating severity. Female novice raters ( n = 28) from a demographically homogeneous background were recruited to rate on two occasions essays written by 111 students in an intermediate-level Chinese as a foreign language program. Raters’ personality traits were measured using the complete version of NEO-PI-R. Many-faceted Rasch measurement model and repeated measurement were applied to yield more robust estimates of rating severity. In addition, rating order effect was carefully controlled. Extroversion was found to be positively correlated with severity, r(26) = .495, p = .010. Furthermore, Extroversion was found to be a valid predictor of severity, t(24) = 2.792, p = .010, R2 = .21, Cohen’s d = .77, Hattie’s r = .37. Practical implications for developing more individualized online rater calibration for large-scale writing assessments were discussed, followed by limitations of the present study.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Literature Reviewmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Methodologically Improved Study on Raters’ Personality and Rating Severity in Writing Assessment

2021

View full text Add to dashboard Cite

show abstract

“…Much attention has also been paid to analysing the characteristics of raters from a number of different angles. Many scholars and researchers placed raters in the centre of writing performance assessment advocating that raters' educational background, language proficiency, rater experience and expertise in scoring, as well as personal traits were crucial to the objectivity of measuring writing skills (Johnson & Lim, 2009;Bijani, 2010;Lim 2011;Alaei et al, 2014). Lim ( 2011) provided the evidence of a special role of rater training and its impact on novice and experienced raters' scoring and its quality (p. 543).…”

Section: Literature Reviewmentioning

confidence: 99%

Investigation of Writing Assessment Literacy of Ukrainian University Teachers

Kvasova

Kavytska

Osidak

2019

ald

View full text Add to dashboard Cite

The rating process of students’ writing has been a long-standing concern in L2 large-scale standardized and classroom-based assessment. Several studies have tried to identify how the raters make scoring decisions and assign scores to ensure validity of writing assessment. The current paper addresses the issue of writing assessment practices of Ukrainian university teachers, how they approach rating scales and criteria with an attempt to understand culturally specific challenges of teachers’ writing assessment in Ukraine. To investigate the issue, this study employs the analysis of the survey results obtained from 104 university teachers of English. The survey consisted of 13 questions that provided insight into current practices in assessment of writing, such as frequency of assessment, use of rating scales, rater’s profile, criteria of assessment, feedback and rewriting, training in assessment of writing. The survey responses show that assessment in Ukraine is not regulated by common standard, and thus the approach to students’ writing assessment is often intuitive. A frequent practice is that teachers tend to rely on errors – as observable features of the text – to justify their rating decisions, Consequently, by shifting focus onto the surface features of writing, grammar mistakes in particular, the teachers underrate such criteria as “register”, “compliance with textual features” and “layout”. Additionally, the data reveal contradictory findings about writing assessment literacy of the teachers questioned. Even though most teachers claim they apply scales while rating, many confess they cannot tell the difference between holistic and analytic scales. Moreover, the results indicate that feedback is not yet a meaningful interaction between a Ukrainian teacher and a learner. Therefore, the results of the study demonstrate the need for the improvement in writing assessment practices, which could be achieved through providing training and reorientation to help Ukrainian teachers develop common understanding and interpretation of task requirements and scale features.

show abstract

“…It benefits the raters as it provides an overview of students' achievement (Lane, 1996). From a practicality viewpoint, it is faster (Alaei, Ahmadi, & Zadeh, 2014) and absolutely saves time by decreasing the number of judgment the raters make (Jackson, 2017).…”

mentioning

confidence: 99%

Developing Scoring Rubric: Do the Students Need It?

Hima

Saputro

2017

ELT echo

View full text Add to dashboard Cite

For many reasons, the use of scoring rubrics has acquired more attention. However, the number of studies investigating the perception and the writing performance as well as the relationship between those two variables in the use of rubric is limited. Therefore, by involving 27 students in Writing-class of an English Language Education Department at a particular university in Indonesia, the present study aimed to explore the students' perceptions through surveys and their writing performance in the three different situations in which they had no scoring rubric, generate their own scoring rubric and had teachermade scoring rubric prior to the writing tasks. Also, the study was intended to explore how the students perceived the three situations related to their writing performance. After analyzing the data, it was found that the most helpful situation for the students was the second situation. It also helped the students to achieve the higher score compared to the other two situations. However, there was a contradictory finding in which the students' perception and their performance were negatively correlated.

show abstract

The Impact of Rater's Personality Traits on Holistic and Analytic Scores: Does Genre Make any Difference too?

Cited by 4 publications

References 14 publications

A Methodologically Improved Study on Raters’ Personality and Rating Severity in Writing Assessment

A Methodologically Improved Study on Raters’ Personality and Rating Severity in Writing Assessment

Investigation of Writing Assessment Literacy of Ukrainian University Teachers

Developing Scoring Rubric: Do the Students Need It?

Contact Info

Product

Resources

About