“It Is Just a Flu”: Assessing the Effect of Watch History on YouTube’s Pseudoscientific Video Recommendations

Abstract

The role played by YouTube's recommendation algorithm in unwittingly promoting misinformation and conspiracy theories is not entirely understood. Yet, this can have dire real-world consequences, especially when pseudoscientific content is promoted to users at critical times, such as the COVID-19 pandemic. In this paper, we set out to characterize and detect pseudoscientific misinformation on YouTube. We collect 6.6K videos related to COVID-19, the Flat Earth theory, as well as the anti-vaccination and anti-mask movements. Using crowdsourcing, we annotate them as pseudoscience, legitimate science, or irrelevant and train a deep learning classifier to detect pseudoscientific videos with an accuracy of 0.79. We quantify user exposure to this content on various parts of the platform and how this exposure changes based on the user's watch history. We find that YouTube suggests more pseudoscientific content regarding traditional pseudoscientific topics (e.g., flat earth, anti-vaccination) than for emerging ones (like COVID-19). At the same time, these recommendations are more common on the search results page than on a user's homepage or in the recommendation section when actively watching videos. Finally, we shed light on how a user's watch history substantially affects the type of recommended videos.

Auditing YouTube’s Recommendation Algorithm for Misinformation Filter Bubbles

“…More specifically, we observe worse situation regarding the topics of vaccination and (partially) 9/11 conspiracies and some improvements (less misinformation) for moon landing or chemtrails conspiracies. In addition, we replicated, to a lesser extent, the works of Hou et al [26] and Papadamou et al [38]. We reused their classification models, which we adapted and applied on our collected data.…”

“…This lack has been partially compensated by external black-box auditing studies performed by the researchers, such as [1,4,27,38,41,48] that aimed to quantify the portion of misinformative content being recommended on social media platforms. With respect to YouTube, which is the subject of the audit presented in this paper, previous works investigated how a user can enter a filter bubble.…”

“…With respect to YouTube, which is the subject of the audit presented in this paper, previous works investigated how a user can enter a filter bubble. Multiple studies demonstrated that watching a series of misinformative videos strengthens the further presence of such content in recommendations [1,27,38], or that following a path of the "up next" videos can bring the user to a very dubious content [48]. However, no studies have covered if, how or with what "effort" can the user "burst" (lessen) the bubble.…”

“…We reused their classification models, which we adapted and applied on our collected data. Finally, we used the better performing Papadamou's model [38] to predict the labels for the videos that were not labeled manually.…”

“…Section 4.7), we do not publish the labels predicted by our trained machine learning model, only their aggregated numbers necessary to reproduce the results. Nevertheless, since we train and use models published in prior works [26,38], it is also possible to replicate this part of our study. In addition, we do not publish copyrighted data (video title, description, etc.).…”