The inter-rater reliability of university students' evaluations of teaching quality was examined with cross-classified multilevel models. Students (N = 480) evaluated lectures and seminars over three years with a standardised evaluation questionnaire, yielding 4224 data points. The total variance of these student evaluations was separated into the variance components of courses, teachers, students and the student/teacher interaction. The substantial variance components of teachers and courses suggest reliability. However, a similar proportion of variance was due to students, and the interaction of students and teachers was the strongest source of variance. Students' individual perceptions of teaching and the fit of these perceptions with the particular teacher greatly influence their evaluations. This casts some doubt on the validity of student evaluations as indicators of teaching quality and suggests that aggregated evaluation scores should be used with caution.

No abstract

Validity is an important issue when measuring teaching quality using student evaluations. This study examined the effects of psychology students' subjective clarity about study contents and their prior interest in the subject as variables possibly biasing the evaluations of psychology courses. German students (N ¼ 292) evaluated lectures and seminars over five years with a standardized questionnaire, yielding 3,348 data points. In cross-classified multilevel models, we separated the total variance into the variance components of course, teacher, student, and the Teacher x Student interaction and found evidence for biasing effects with regard to the students' clarity about study contents and prior subject interest. These effects were small overall and were stronger for lectures than for seminars. These results suggest that the validity of evaluations of teaching in psychology can be improved by creating realistic expectations of what psychology is about before students choose psychology as a study subject.

No abstract