Are Teacher Course Evaluations Biased Against Faculty That Teach Quantitative Methods Courses?

Royal, Kenneth D.; Stockdale, Myrah R.

doi:10.5430/ijhe.v4n1p217

Cited by 14 publications

(9 citation statements)

References 32 publications

(27 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…8 Age, [11], charisma [13], and physical attractiveness [14,15] are also associated with SET. Other factors generally not in the instructor's control that may affect SET scores include class time, class size, mathematical or technical content [16], and the physical classroom environment [17]. Many studies cast doubt on the validity of SET as a measure of teaching effectiveness (see Johnson [4, for a review and analysis, Pounder [18] for a review, and Galbraith et al [19], Carrell and West [2] for exemplars).…”

Section: Discussion Other Studiesmentioning

confidence: 99%

Student evaluations of teaching (mostly) do not measure teaching effectiveness

Boring¹,

Ottoboni²,

Stark

2016

ScienceOpen Research

328

229

View full text Add to dashboard Cite

<p>Student evaluations of teaching (SET) are widely used in academic personnel decisions as a measure of teaching effectiveness. We show:</p><ul> <li>SET are biased against female instructors by an amount that is large and statistically significant</li> <li>the bias affects how students rate even putatively objective aspects of teaching, such as how promptly assignments are graded</li> <li>the bias varies by discipline and by student gender, among other things</li> <li>it is not possible to adjust for the bias, because it depends on so many factors</li></ul><ul> <li>SET are more sensitive to students' gender bias and grade expectations than they are to teaching effectiveness</li> <li>gender biases can be large enough to cause more effective instructors to get lower SET than less effective instructors.</li></ul><p>These findings are based on nonparametric statistical tests applied to two datasets: 23,001 SET of 379 instructors by 4,423 students in six mandatory first-year courses in a five-year natural experiment at a French university, and 43 SET for four sections of an online course in a randomized, controlled, blind experiment at a US university.</p>

show abstract

Section: Discussion Other Studiesmentioning

confidence: 99%

Student evaluations of teaching (mostly) do not measure teaching effectiveness

Boring¹,

Ottoboni²,

Stark

2016

ScienceOpen Research

328

229

View full text Add to dashboard Cite

show abstract

“…Other researchers used Rasch models to identify biases that may threaten the interpretation of SET results. Royal and Stockdale (2015) investigated SET ratings to determine if response biases were detected in courses relating to quantitative methods and approaches. Similarly, Haladyna and Hess (1994) applied a Rasch model to identify and correct biases in data that included faculty member, item rater, gender of rater, rater perception of course type, and whether course was required or elective.…”

Section: Theoretical Frameworkmentioning

confidence: 99%

Psychometric quality of a student evaluation of teaching survey in higher education

Oon¹,

Benson²,

Kam³

2016

Assessment & Evaluation in Higher Education

View full text Add to dashboard Cite

“…In this sense, it has been confirmed that in undergraduate courses, the area of knowledge induces a considerable bias in SETs ( Cashin, 1990 ; Beran & Violato, 2005 ; Centra, 2009 ; Uttl, White & Morin, 2013 ; Royal & Stockdale, 2015 ; Uttl & Smibert, 2017 ; Arroyo-Barrigüete et al, 2021 ). Furthermore, it is also clear that this bias has relevance, regardless of the percentage of the variance explained by that factor.…”

Section: Discussionmentioning

confidence: 88%

“…Nevertheless, the recent work by Uttl & Smibert (2017) highlights that a particularly relevant noninstructional bias, the subject area to which the course belongs, has likely been undervalued in many previous studies. Numerous articles have found a negative bias toward teachers who teach quantitative and/or STEM courses ( Cashin, 1990 ; Beran & Violato, 2005 ; Centra, 2009 ; Uttl, White & Morin, 2013 ; Royal & Stockdale, 2015 ; DeFrain, 2016 ; Rosen, 2018 ; Arroyo-Barrigüete et al, 2021 ), but Uttl and Smibert claim that in several of these articles, the relevance of this bias is underestimated. They stated that parametric statistics are not appropriate due to several factors.…”

Section: Introductionmentioning

confidence: 99%

Spain is not different: teaching quantitative courses can also be hazardous to one’s career (at least in undergraduate courses)

Barrigüete

Obregón

Ortiz-Lozano

et al. 2022

PeerJ

View full text Add to dashboard Cite

Student evaluations of teaching (SETs) have become a widely used tool for assessing teaching in higher education. However, numerous investigations have shown that SETs are subject to multiple biases, one of which is particularly relevant, namely, the area of knowledge to which the subject belongs. This article aims to replicate the article by Uttl & Smibert (2017, https://doi.org/10.7717/peerj.3299) in a different educational context to verify whether the negative bias toward instructors who teach quantitative courses found by the authors in the US also appears in the Spanish university system. The study was conducted at the Business and Law School of the Universidad Pontificia Comillas, a private Spanish university, using two different samples. First, we analyzed undergraduate courses using a sample of 80,667 SETs in which 2,885 classes (defined as a single semester-long course taught by an individual instructor to a specific group of students), 488 instructors, and 322 different courses were evaluated over a time period of four academic years (2016/2017–2019/2020). Second, in the same period, 16,083 SETs corresponding to master’s degree courses were analyzed, which involved the study of 871 classes, 275 instructors, and 155 different courses. All the data included in the analysis were obtained from official university surveys developed by a team of professionals specialized in teaching quality responsible for ensuring the reliability of the information. At the degree level, the results show that despite the considerable cultural and temporal difference between the samples, the results are very similar to those obtained by Uttl & Smibert (2017, https://doi.org/10.7717/peerj.3299); i.e., professors teaching quantitative courses are far more likely to obtain worse SETs than instructors in other areas. There are hardly any differences at the master’s degree level, regardless of whether nearly 75% of master’s degree instructors also teach at the undergraduate level. This leads us to three different conclusions. (1) Evidence suggests that the reason for these differences is not due to faculty teaching quantitative courses being less effective than faculty teaching in some other fields. Our results indicate that the same instructor is evaluated very differently depending on whether he or she teaches at the undergraduate or master’s level. (2) It is essential to avoid comparisons of SETs between different areas of knowledge, at least at the undergraduate level. (3) A significant change in the use and interpretation of SETs is imperative, or its replacement by other evaluation mechanisms should be considered. If this does not occur, it is possible that in the future, there will be an adverse selection effect among professors of quantitative methods; i.e., only the worst professionals in quantitative methods will opt for teaching since the good professionals will prefer other jobs.

show abstract

Are Teacher Course Evaluations Biased Against Faculty That Teach Quantitative Methods Courses?

Cited by 14 publications

References 32 publications

Student evaluations of teaching (mostly) do not measure teaching effectiveness

Student evaluations of teaching (mostly) do not measure teaching effectiveness

Psychometric quality of a student evaluation of teaching survey in higher education

Spain is not different: teaching quantitative courses can also be hazardous to one’s career (at least in undergraduate courses)

Contact Info

Product

Resources

About