Conflict of interest explains the size of student evaluation of teaching and learning correlations in multisection studies: a meta-analysis

Abstract

We examined the associations between the size of student evaluation of teaching and learning (SET/learning) correlations and presence of several conflicts of interest (COIs) including corporate, administrative, evaluation unit, SET author, and funder interests. Our meta-analyses of SET/learning correlations reported by multisection studies show that researchers with a vested interest in finding large positive SET/learning correlations found, on average, large positive SET/learning correlations. In contrast, researchers with no identifiable COIs found that SET/learning correlations were zero or nearly zero. The largest SET/learning correlations were reported by authors with ties to SET selling corporations. Smaller but still substantial SET/learning correlations were reported by researchers with administrative assignments and by researchers in evaluation units/departments responsible for the administration of SET. Moreover, authors with the most significant COIs were publishing their studies primarily prior to 1981 whereas authors with no or less significant COIs were publishing their studies in 1981 or afterwards. Studies published prior to 1981 reported small but significant (r = .31) SET/learning correlations whereas studies published in 1981 and after reported near zero, non-significant SET/learning correlations (r = .06). The presence of COIs was associated with earlier publication date but also with smaller samples. Finally, whereas corporate, administrative, and evaluation unit authors nearly ceased publishing multisection studies on SET/learning correlations, authors from business and economics departments are now responsible for the substantial portion of newer, larger, and higher quality studies published in 1981 and after.

Student Evaluations of Teaching Encourages Poor Teaching and Contributes to Grade Inflation: A Theoretical and Empirical Analysis

“…As Upton Sinclair (1934Sinclair ( / 1940) once wrote, "It is difficult to get a man to understand something, when his salary depends on his not understanding it" (p. 109). The evidence presented by Uttl et al (2019) is persuasive. They found that SET-learning correlations were much larger when at least one author of a study was associated with an SET corporation.…”

“…As Clayson (2009) reported, the size of SET-learning correlations were strongly correlated with year of publication. When Uttl et al (2019) looked in more detail at this association, they found that the 69 studies published prior to 1981 yielded an average correlation of r ¼ .31, whereas the 28 studies published in or after 1981 had an average correlation of r ¼ .06. This is a substantial difference, and one wonders how such a difference can be explained.…”

“…This is a substantial difference, and one wonders how such a difference can be explained. Uttl et al (2019) offered conflict of interest (COI) as an explanation: Many of the early studies were published by researchers who either worked for or owned corporations that sell SET systems and thus had a (financial) interest in finding substantial SET-learning correlations. There is ample evidence from pharmaceutical research that funding sources influence research outcomes (e.g., Bekelman et al, 2003;Lundh et al, 2018).…”

Video Improves Learning in Higher Education: A Systematic Review

“…Quasi-experimental studies are more likely to be confounded, particularly if students could choose the mode of delivery that suited them best. We reviewed the effects on learning, rather than student satisfaction, because learning is a key goal of all universities, and student evaluations are poor predictors of educational attainment (Uttl et al, 2017, 2019). Overall, in this review we aimed to determine (a) what are the learning effects of swapping other content for videos and (b) what are the effects of adding videos to existing course materials?…”

Students’ Evaluation of Teaching and Their Academic Achievement in a Higher Education Institution of Ecuador

“…However, taking into consideration our results and the results of previous meta-analyses, especially the comprehensive metaanalysis of Uttl et al (2017), the influence of SET on academic achievement seems to be sufficiently limited to make relevant administrative decisions. Although use of SET as a feedback for teachers' use and as a measure of student satisfaction is not problematic (Spooren et al, 2013;Uttl et al, 2017), the use of SET as a measure of teachers' effectiveness for making administrative decisions about teachers' hiring, firing, promotions, and merit pay is controversial (Uttl et al, 2017(Uttl et al, , 2019American Sociological Association, 2019).…”