Effects of the semantic similarity of items on student ratings of instructors.

Abstract

Factor analysis of student ratings of teacher effectiveness may reveal more about the semantic similarity of items than it does about actual covariation in instructor behavior. To test this hypothesis, 28 graduate students were asked to rate several hypothetical instructor profiles, constructed by systematically manipulating information about the instructors' classroom behaviors. Factor analyses of the student ratings were performed before and after all behavioral information in the instructor profiles had been statistically removed. The results revealed no substantive change in the underlying factor structure. In both analyses the factors represented clusters of semantically equivalent items. Thus, students appeared to be imposing an implicit semantic organization on their ratings, apart from any covariance among the instructors' actual classroom behavior.

Factorial Validity of Student Ratings of Instruction

“…Thus, the structure of personality factors could be influenced by what is thought to go with what rather than what actually goes with what. Cadwell and Jenkins (1985) present a clear example of the possible effect of systematic distortion on student ratings of teaching:…”

“…The effects of systematic distortion or conceptual association bias have been studied in several different areas, including general personality assessment (Berman and Kenny, 1976;Block, Weiss, and Thorne, 1979;Hamilton and Rose, 1980;King and Koehler, 2000;Paunonen and Jackson, 1979;Riemann and Angleitner, 1993;Shweder, 1975), clinical applications (Chadwick and Taylor, 2000;Chapman, 1967, 1969;Dollinger, Greening, and Radtke, 2001;Shweder, 1977;Shweder and D'Andrade, 1980), job performance evaluations (Cooper, 1981;Dewhirst and Berman, 1978;Kozlowski, Kirsch, and Chao, 1986;Vonk, 1998), and student ratings of instructors (Cadwell and Jenkins, 1985;Lodzinski, 1991;Lodzinski, Schneider, and Murray, 1991;Whitely and Doyle, 1976). Unfortunately, there appears to be a lack of consensus on the extent to which covariance in ratings of dimensions of personality or performance is biased by systematic distortion.…”

Interpersonal Competence Development of University Students -Exploring a Social Problem-Solving Measurement Approach

“…In many cases, attitude-based rating scales produced good reliability and validity. However, some researchers argued that such results could depend on the semantic similarity of items more than on variations in real behavior (e.g., Cadwell & Jenkins, 1985), or that performance judgements are influenced not only by reality, but also by implicit standards of effective behavior (e.g., Kishor, 1995). Additionally, in social research, some researchers found evidence for an "attitude-behavior gap" which means that sometimes or even often people do not behave according to their attitudes, values, or intentions (e.g., Greve, 2001).…”

Information utilization in performance rating: Interactive effects of purpose and cue dimensionality

“…Hypothetical ratee profiles limited the amount of detail that could be included, but were judged to be suitable because it is a procedure that allows control over the amount of information. Moreover, the use of hypothetical profiles was justified because in a real situation students often evaluate on general impressions and not on specific details (Cadwell & Jenkins, 1985).…”

“…This choice was based on the wide use of policy-capturing analysis in previous related research (Cadwell & Jenkins, 1985;Zedeck & Cascio, 1982;Zedeck & Kafry, 1977), and in various other areas where information utilization has been of interest (Borko & Cadwell, 1982;Cadwel & Jenkins, 1986).…”