Auditing Partisan Audience Bias within Google Search

Abstract

There is a growing consensus that online platforms have a systematic influence on the democratic process. However, research beyond social media is limited. In this paper, we report the results of a mixed-methods algorithm audit of partisan audience bias and personalization within Google Search. Following Donald Trump's inauguration, we recruited 187 participants to complete a survey and install a browser extension that enabled us to collect Search Engine Results Pages (SERPs) from their computers. To quantify partisan audience bias, we developed a domain-level score by leveraging the sharing propensities of registered voters on a large Twitter panel. We found little evidence for the "filter bubble'' hypothesis. Instead, we found that results positioned toward the bottom of Google SERPs were more left-leaning than results positioned toward the top, and that the direction and magnitude of overall lean varied by search query, component type (e.g. "answer boxes"), and other factors. Utilizing rank-weighted metrics that we adapted from prior work, we also found that Google's rankings shifted the average lean of SERPs to the right of their unweighted average.

How search engines disseminate information about COVID-19 and why they should do better

“…To account for differences in search result ranking, we used RBO, a metric designed explicitly for investigating similarities in the output of search engines (Webber et al, 2010;Cardoso & Magalhães, 2011;Robertson et al, 2018). Unlike JI, RBO weights top search results more than lower ones to take into account that a source's ranking influences the probability of the user seeing it (Cutrell & Guan, 2007).…”

Search as News Curator

“…These studies expose a range of methodological challenges in search engine audits that we consider, including the choice of search terms, language settings, geolocation, search history, and logged-in status [53]. While some recent studies have emphasized ecological validity by leveraging plugins which scrape search results from real users' results [17,54,55,56], here we opt for the control, consistency, and scale afforded by automated means of scraping results.…”

Who's the Guinea Pig?

“… Robertson et al report that their scores have high correlation with the other scoring systems mentioned here[61].…”

“…In our dataset, we observe many overtly political websites running experiments using Optimizely. To quantify this, we adopt a mapping of websites to partisan bias scores developed by Robertson et al [61]. These scores are calculated based on the relative frequency that a given website is shared on Twitter by registered Democrats and Republicans.…”