Rating of statements in an internal course evaluation: comparison between different groups of assessors

Abstract

The increased concern for accountability and quality improvement is inducing universities throughout Europe to evaluate and control performance. The establishment of criteria for educational quality, a requirement for a successful evaluation, was studied as part of an evaluation of the dental education programme at the Catholic University of Leuven. Criteria suggested by different groups (members of the internal evaluation commission including graduate and postgraduate students; members of the academic staff of the School of Dentistry, Oral Pathology and Maxillo-facial Surgery, and other members of the academic staff, not directly belonging to the school), were analysed by means of ANOVA with Scheffé posthoc tests to identify intergroup differences. It was hypothesized that a closer connection to the dental curriculum or a higher responsibility in the evaluation process would result in stricter criteria. For 108 different questions (in the form of statements with a six point Likert-type scale), members of the three groups had to indicate the score below which the quality would be considered suboptimal. Except for one question, no significant differences were found between ratings given by the members of the internal evaluation commission and other members of the School. Significant differences were found between non-school members and school- and/or commission-members. This study indicates that agreement on educational quality may be presupposed within an academic faculty and between members of the internal evaluation commission. However, the study also indicated that some caution should be exercised when 'external' teachers are included in the evaluation process.