Exploring Variability Sources in Student Evaluation of Teaching via Many-Facet Rasch Model

Abstract

Evaluating quality of teaching is important in nearly every higher education institute. The most common way of assessing teaching effectiveness takes place through students. Student Evaluation of Teaching (SET) is used to gather information about students' experiences with a course and instructor's performance at some point of semester. SET can be considered as a type of rater mediated performance assessment where students are the raters and instructors are the examinees. When performance assessment becomes a rater mediated assessment process, extra measures need to be taken into consideration in order to create more reliable and fair assessment practices. The study has two main purposes; (a) to examine the extent to which the facets (instructor, student, and rating items) contribute to instructors' score variance and (b) to examine the students' judging behavior in order to detect any potential source of bias in student evaluation of teaching by using the Many-Facet Rasch Model. The data set includes one thousand 235 students' responses from 254 courses. The results show that a) students greatly differ in the severity while rating instructors, b) students were fairly consistent in their ratings, c) students as a group and individual level are tend to display halo effect in their ratings, d) students are clustered at the highest two categories of the scale and e) the variation in item measures is fairly low. The findings have practical implications for the SET practices by improving the psychometric quality of measurement.Keywords: Student evaluation of teaching, Many Facet Rasch Model, psychometric analysis Öz Öğretim niteliğini değerlendirmek neredeyse her yükseköğretim kurumlarında önemlidir. Öğretimin etkinliğini değerlendirmenin en yaygın yöntemi öğrenciler üzerinden gerçekleştirilmektedir. Ders değerlendirme anketi (DDA) yoluyla dönemin herhangi bir zamanında öğrencilerin ders ve öğretim elemanı hakkındaki görüş ve tecrübelerine ilişkin bilgi toplanır. DDA, puanlayıcı aracılı bir tür performans değerlendirmesi olarak kabul edilebilir. Bu kez öğrenciler puanlayıcı, öğretim elemanları ise değerlendirilendir. Performans değerlendirme, bir puanlayıcı aracılı değerlendirme süreci olduğunda, ekstra önlemler, daha güvenilir ve adil bir değerlendirme uygulamaları oluşturmak için dikkate alınması gerekir. Bu çalışmanın amacı, a) öğretim elemanlarının puanlarındaki farklılığa/varyansa, puanlama sürecindeki yüzeylerin (öğretim elemanı, öğrenci ve değerlendirme maddeleri) ne derece katkı sağladığını ve b) öğrencilerin yaptiklari puanlamalarda yanlılığa yol açacak potansiyel kaynakları çok yüzeyli Rasch modeli yardımıyla incelemektir. Çalışmada kullanılan veri seti 254 dersten 1.235 öğrencinin değerlendirmelerini kapsamaktadır. Sonuçlara göre a) öğrenciler, öğretim elemanlarını değerlendirirken farklı katılık/cömertlik derecesi göstermektedirler, b) çoğu öğrenci kendi içinde oldukça tutarlı, c) grup olarak, değerlendirmelerde halo etkisi oldugu görülmektedir, d) öğrenciler beşli ölçeğin üst puanlarında kümelenm...

Item and rater variabilities in students’ evaluation of teaching in a university in Ghana: application of Many-Facet Rasch Model

“…This result agrees with the view of Eckes (2015) who argued that rather than operating on collective grounds, student-raters regularly seem to considerably differ regarding deeply fixed, more or less individualised rating predispositions and thereby threatening the validity of the evaluation outcomes. A similar result was also found in a study conducted by Börkan (2017) who also used MFRM to explore the sources of variations in students' appraisal of teaching in a university in Turkey. In Börkan's study, it was also revealed that a larger proportion of the students rated leniently.…”

“…It is not always so that the lecturer has created a positive or negative impression for the students, but some raters (students) may just fail to distinguish between the exact behaviours being measured and other behaviours of the lecturer unrelated to the tagreted behaviours. These behaviours of students which constituted the halo effect were also found among university students in Börkan's (2017) study. The consistency in the results of this study and Börkan's research speaks to the fact that university students, irrespective of their location, have similar behaviours when rating their lecturers.…”

“…It is in very few instances that studies explored the validity and reliability of students' evaluation of teaching using the GT approach (e.g., Feistauer and Richter, 2016 ; Li et al., 2018 ; Quansah, 2020 ; VanLeeuwen et al., 1999 ). A single study, however, investigated the dependability of students’ appraisal of courses and teaching through the lens of MFRM ( Börkan, 2017 ).…”

“…Whereas analyses in CMT directly utilise the raw scores from the dataset, facet/factor measurements are converted to a wide metric scale, known as logit, in the MFRM ( Linacre, 1989 ). Similarly, studies conducted using the GT approach, as compared to the MFRM, also do not provide a detailed picture of the accuracy of students' appraisal of courses and teaching ( Börkan, 2017 ; Linacre, 2003 ). GT offers an overall summary of group-specific analysis of the variances influencing the scores for all objects of measurement (e.g., lecturers, as in this study) ( Linacre, 1994 ).…”

Bias in Student Ratings of Instruction: A Systematic Review of Research from 2012 to 2021

“…Conversely, Socha (2013) found that prior interest level was negatively associated with SRI ratings, possibly mediated by overly high course expectations. Further, students provided higher ratings on course evaluations in elective (vs required) courses, which could also be attributed to their understanding and prior interest in the course (Al-Maamari, 2015;Borkan, 2017;Liu, 2012;E. Park & Dooris, 2020;Radchenko, 2020).…”