The Issue of Proxies and Choice Architectures. Why EU Law Matters for Recommender Systems

Abstract

Recommendations are meant to increase sales or ad revenue, as these are the first priority of those who pay for them. As recommender systems match their recommendations with inferred preferences, we should not be surprised if the algorithm optimizes for lucrative preferences and thus co-produces the preferences they mine. This relates to the well-known problems of feedback loops, filter bubbles, and echo chambers. In this article, I discuss the implications of the fact that computing systems necessarily work with proxies when inferring recommendations and raise a number of questions about whether recommender systems actually do what they are claimed to do, while also analysing the often-perverse economic incentive structures that have a major impact on relevant design decisions. Finally, I will explain how the choice architectures for data controllers and providers of AI systems as foreseen in the EU's General Data Protection Regulation (GDPR), the proposed EU Digital Services Act (DSA) and the proposed EU AI Act will help to break through various vicious circles, by constraining how people may be targeted (GDPR, DSA) and by requiring documented evidence of the robustness, resilience, reliability, and the responsible design and deployment of high-risk recommender systems (AI Act).

Everything is varied: The surprising impact of instantial variation on ML reliability

“…In this article we studied the impact of IV, an often neglected type of uncertainty affecting data (as representations and proxies [67] of entities' features and behaviors), on the performance and robustness of ML models. Through a realistic experiment on COVID-19 diagnosis, a problem of significant practical interest which we take as paradigmatic of the class of applications with high risk and impact on human subjects, we showed how standard ML algorithms can be strongly impacted by the presence of IV, failing to generalize properly.…”

Reviews in Recommender Systems: 2022

“…The latter describes algorithms which use some or all protected characteristics in their data to make decisions. Blind algorithms not only tend to underperform (Žliobaitė and Custers, 2016;Bent, 2019;Xiang and Raji, 2019;Wachter et al, 2020;Kim, 2022), but also often learn spurious correlations in the data that can serve as proxies for the protected characteristics in their datasets, which can equally lead to discrimination cases (Žliobaitė and Custers, 2016;Chander, 2017;Kim, 2017;Bent, 2019;Wachter et al, 2020;Adams-Prassl, 2022;Hildebrandt, 2022). This can occur, for instance, when a correlation between ethnicity and postal code is drawn or when the recommendation algorithm unintentionally incorporates implicit gender information from interaction data because of different preferences of male and female users .…”

Fairness of recommender systems in the recruitment domain: an analysis from technical and legal perspectives

“…The latter describes algorithms which use some or all protected characteristics in their data to make decisions. Blind algorithms not only tend to underperform (Žliobaitė and Custers, 2016;Bent, 2019;Xiang and Raji, 2019;Wachter et al, 2020;Kim, 2022), but also often learn spurious correlations in the data that can serve as proxies for the protected characteristics in their datasets, which can equally lead to discrimination cases (Žliobaitė and Custers, 2016;Chander, 2017;Kim, 2017;Bent, 2019;Wachter et al, 2020;Adams-Prassl, 2022;Hildebrandt, 2022). This can occur, for instance, when a correlation between ethnicity and postal code is drawn or when the recommendation algorithm unintentionally incorporates implicit gender information from interaction data because of different preferences of male and female users (Ganhör et al, 2022).…”