Auditing Source Diversity Bias in Video Search Results Using Virtual Agents

Urman, Aleksandra; Makhortykh, Mykola; Ulloa, Roberto

doi:10.1145/3442442.3452306

Cited by 10 publications

(13 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Diversity in search engine results. Urman et al [27] track several search engines, to audit source diversity and search concentration. This is achieved by submitting a static list of 62 keywords.…”

Section: The Bobby Audit Formmentioning

confidence: 99%

See 1 more Smart Citation

Algorithmic audits of algorithms, and the law

Merrer¹,

Pons²,

Trédan³

2022

Preprint

View full text Add to dashboard Cite

Algorithmic decision making is now widespread, ranging from health care allocation to more common actions such as recommendation or information ranking. The aim to audit these algorithms has grown alongside. In this paper, we focus on external audits that are conducted by interacting with the user side of the target algorithm, hence considered as a black box. Yet, the legal framework in which these audits take place is mostly ambiguous to researchers developing them: on the one hand, the legal value of the audit outcome is uncertain; on the other hand the auditors' rights and obligations are unclear.The contribution of this paper is to articulate two canonical audit forms to law, to shed light on these aspects:• the first audit form (we coin the Bobby audit form) checks a predicate against the algorithm, while the second (Sherlock) is more loose and opens up to multiple investigations. We find that: Bobby audits are more amenable to prosecution, yet are delicate as operating on real user data. This can lead to reject by a court (notion of admissibility). Sherlock audits craft data for their operation, most notably to build surrogates of the audited algorithm. It is mostly used for acts for whistleblowing, as even if accepted as a proof, the evidential value will be low in practice.• these two forms require the prior respect of a proper right to audit, granted by law or by the platform being audited; otherwise the auditor will be also prone to prosecutions regardless of the audit outcome. This article thus highlights the relation of current audits with law, in order to structure the growing field of algorithm auditing.Society rules through law, so that law is supposed to hold service providers and their algorithms accountable. In particular, decision-making algorithms are now widespread [6]. They directly face users, and govern large portions of our lives (from apparently subtle decisions such as recommendations, to more life changing ones such criminal justice or health care allocation [14]). The legal perspective on algorithms, especially of online platforms, is evolving to strong legal frameworks. For example, at an european level, in order to protect the fundamental rights of european residents 1 , or to frame artificial intelligence systems 2 explicitly shows the willingness to better regulate algorithms.The IT perspective and the nascent field of audits. As computer scientists and engineers, we are used to design and develop algorithms that process information and that can have an important impact on society [11,19]. For fine tuning these, developers operate a controlled feedback loop on data fed as inputs to the algorithm, and the corresponding algorithm results (output accuracy for instance).Considering an exterior viewpoint (the viewpoint of users or regulators), that observes or audits the behavior of remote algorithms is less frequent. A so called black box approach on algorithms can be dated back to Moore's tests black box automata in 1956 [20]. Relatively recent and sporadic works instead placed this ...

show abstract

Section: The Bobby Audit Formmentioning

confidence: 99%

“…26 Proposal for a REGULATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL on a Single Market For Digital Services (Digital Services Act) and amending Directive 2000/31/EC COM/2020/825 final. 27 Called "recipient of the service" in the european proposal. 28 Article 19 of the digital services act proposal.…”

Section: 42mentioning

confidence: 99%

Algorithmic audits of algorithms, and the law

Merrer¹,

Pons²,

Trédan³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Coverage (and visibility) has not directly been addressed in news search scholarship because of the reactivity of search engines, namely, search engines do not feature a selection of materials per se (e.g., as in a news website), but retrieve them in response to user queries. For any query, search engines return a long list of news articles, albeit in the majority of cases individuals will interact with only those at the top ( Pan et al, 2007 ; Urman & Makhortykh, 2021 ). Because the relevance of news items changes over time, more relevant items can appear at the top when the individual searches again.…”

Section: Introductionmentioning

confidence: 99%

Novelty in News Search: A Longitudinal Study of the 2020 US Elections

Ulloa

Makhortykh

Urman

et al. 2023

Social Science Computer Review

Self Cite

View full text Add to dashboard Cite

The 2020 US elections news coverage was extensive, with new pieces of information generated rapidly. This evolving scenario presented an opportunity to study the performance of search engines in a context in which they had to quickly process information as it was published. We analyze novelty, a measurement of new items that emerge in the top news search results, to compare the coverage and visibility of different topics. Using virtual agents that simulate human web browsing behavior to collect search engine result pages, we conduct a longitudinal study of news results of five search engines collected in short bursts (every 21 minutes) from two regions (Oregon, US and Frankfurt, Germany), starting on election day and lasting until one day after the announcement of Biden as the winner. We find more new items emerging for election related queries (“joe biden,” “donald trump,” and “us elections”) compared to topical (e.g., “coronavirus”) or stable (e.g., “holocaust”) queries. We demonstrate that our method captures sudden changes in highly covered news topics as well as multiple differences across search engines and regions over time. We highlight novelty imbalances between candidate queries which affect their visibility during electoral periods, and conclude that, when it comes to news, search engines are responsible for such imbalances, either due to their algorithms or the set of news sources that they rely on.

show abstract

“…One of the most studied platforms are web search engines -almost half of the auditing works reviewed by Bandy [5] were focused on Google alone -as a plethora of concerns have been raised about representation, biases, copyrights, transparency and discrepancies in their outputs. Research has analysed issues in areas such as elections [6][7][8][9][10][11], filter bubbles [12][13][14][15][16][17], personalised results [18,19], gender and race biases [20][21][22], health [23][24][25], source concentration [10,[26][27][28][29], misinformation [30], historical information [31,32] and dependency on user-generated content [33,34].…”

Section: Introductionmentioning

confidence: 99%

Scaling up search engine audits: Practical insights for algorithm auditing

Ulloa

Makhortykh

Urman

2022

Journal of Information Science

Self Cite

View full text Add to dashboard Cite

Algorithm audits have increased in recent years due to a growing need to independently assess the performance of automatically curated services that process, filter and rank the large and dynamic amount of information available on the Internet. Among several methodologies to perform such audits, virtual agents stand out because they offer the ability to perform systematic experiments, simulating human behaviour without the associated costs of recruiting participants. Motivated by the importance of research transparency and replicability of results, this article focuses on the challenges of such an approach. It provides methodological details, recommendations, lessons learned and limitations based on our experience of setting up experiments for eight search engines (including main, news, image and video sections) with hundreds of virtual agents placed in different regions. We demonstrate the successful performance of our research infrastructure across multiple data collections, with diverse experimental designs, and point to different changes and strategies that improve the quality of the method. We conclude that virtual agents are a promising venue for monitoring the performance of algorithms across long periods of time, and we hope that this article can serve as a basis for further research in this area.

show abstract

Auditing Source Diversity Bias in Video Search Results Using Virtual Agents

Cited by 10 publications

References 24 publications

Algorithmic audits of algorithms, and the law

Algorithmic audits of algorithms, and the law

Novelty in News Search: A Longitudinal Study of the 2020 US Elections

Scaling up search engine audits: Practical insights for algorithm auditing

Contact Info

Product

Resources

About