Crossed Random-Effect Modeling: Examining the Effects of Teacher Experience and Rubric Use in Performance Assessments

Abstract

Problem Statement: Performance assessments have emerged as an alternative method to measure what a student knows and can do. One of the shortcomings of performance assessments is the subjectivity and inconsistency of raters in scoring. A common criticism of performance assessments is the subjective nature of scoring procedures. The effectiveness of the performance assessment procedure depends highly on the quality and coordination of teacher and rubric. To gain a better understanding of the interaction between teachers and performance assessments, it is crucial to examine the effects of teacher-related factors and how teachers interact with scoring rubrics when grading performance assessments. One of these factors is teachers' work and scoring experience. When grading performance assessments, the experienced teachers may be expected to grade student performances more objectively through their experience in instruction and evaluation than the teachers with less teaching and scoring experience.

The validity and inter-rater reliability of project assessment in mathematics learning

“…It underlies that project task will help students develop their collaboration, communication, and problem-solving abilities (Shariff, Johan, & Jamil, 2013). Those abilities are the keys needed in global life (Kan & Bulut, 2014).…”

“…In other words, the teacher's ability to understand and apply the assessment rubric as well as teacher subjectivity level is very influential in giving the assessment. For this reason, Kan and Bulut (2014) suggest two things that can be done to minimize the subjectivity effect on project assessment. First, developing a clear scoring mechanism which contains a description of the project task to be done by students and teachers.…”

“…Referring to Kan and Bulut (2014), the effectiveness of the project assessment depends on the quality and coordination between the teacher and the rubric. In other words, the role of rubrics in assessment context is very essential.…”

“…Some studies (e.g., Smit, Bachmann, Blum, Birri, & Hess, 2017;Wulan, 2008) show that raters or teachers are often inconsistent in using rubrics. It is due to teachers' lack of experiences in using the rubric as well as the quality of the rubric (Kan & Bulut, 2014). Besides, the inconsistency of using the rubric also occurs because of a lack of understanding of rubric's constructs or aspects.…”

“…The aforementioned researches (e.g., Smith et al, 2004;Jonsson & Svingby, 2007;Wulan, 2008) and the experts' opinions (e.g., Kan & Bulut, 2014) indicate the importance of instruments on project assessment along with good rubrics. Thus, teachers who use project assessment can provide relatively similar scores.…”