On Bias in Social Reviews of University Courses

Abstract

University course ranking forums are a popular means of disseminating information about satisfaction with the quality of course content and instruction, especially with undergraduate students. A variety of policy decisions by university administrators, instructional designers and teaching staff affect how students perceive the efficacy of pedagogies employed in a given course, in class and online. While there is a large body of research on qualitative driving factors behind the use of academic rating sites, there is little investigation of the (potential) implicit student bias on said forums towards desirable course outcomes at the institution level.To that end, we examine the connection between course outcomes (student-reported GPA) and the overall ranking of the primary course instructor, as well as rating disparity by nature of course outcomes, for several hundred courses taught at Virginia Tech based on data collected from a popular academic rating forum. We also replicate our analysis for several public universities across the US. Our experiments indicate that there is a discernible albeit complex bias towards course outcomes in the professor ratings registered by students.

Exploring the context of course rankings on online academic forums

“…These studies generally report this correlation to be modest, somewhere between +0.1 and +0.3 [10]. Our preliminary inquiry at the institution level demonstrates that this correlation often matches and sometimes exceeds said figures [14]. The aforementioned work notes that given the learning acquired by students during the course of a given Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.…”

Fine-grained Main Ideas Extraction and Clustering of Online Course Reviews

“…We plan to deploy this pipeline in real-life classes and create teaching assistantgenerated gold-standard summaries [23] for evaluation of algorithm-generated idea coverage against human readers' perceptions. Notably, our approach aims to extract information from course reviews, while we highlight that intrinsic biases in course evaluations do exist [13,4]. We encourage researchers in this field to build upon our method to detect and mitigate biases in course evaluations.…”

Learning to Trust: Understanding Editorial Authority and Trust in Recommender Systems for Education

“…It is worthwhile noting that AV-faculty members do not seek a fully cooperative model of student feedback to solve this challenge in the manner of A-faculty. Frequent reasons posited by faculty include perceived historical patterns of student disengagement and exclusive attention by students to course outcomes, or the outcome bias [12]. This bias is also known to affect student perceptions of instructors on academic social forums like Koofers [9] and RateMyProfessor Another faculty member talked about the requirements of professional diplomas -dictated by the rapidly evolving demands of the job market -as one key driver of student disengagement.…”

“…For our study, user groups include faculty, teaching assistants, and students. signals of trust, for instance, the RS user's interactional awareness of their local neighborhood, or some consensus of the preferences of their self-reported, trustworthy friends in a social network at large [21][13] [12]. While algorithmic awareness of a user's neighborhood is important for producing accurate recommendations, real life recommendation tasks often involve user groups with differences in institutional, group, or task-based roles, powers and prerogatives.…”