Human-First, Please: Assessing Citizen Views and Industrial Ambition for Emotional AI in Recommender Systems

Abstract

This paper qualitatively explores the views of diverse members of the British public on applications of biometric emotional AI technologies patented by two globally dominant consumer-facing recommender systems, Amazon and Spotify. Examining Amazon and Spotify patents for biometric profiling of users’ emotions, disposition, and behaviour to offer them tailored services, ads, and products from their wider platforms, this paper points to industrial ambition regarding emotional AI. Little is known about ordinary people’s views on deployment of such technology, and given the complex, abstract, and future-facing nature of such technologies, ascertaining informed lay views is hard. We address this through our innovative, qualitative study of diverse British-based adults (n=46) that presents to them near-horizon use cases in an interactive fictional narrative that deploys design fiction principles and ContraVision techniques. We find the themes of “usefulness,” “resignation,” “uneasy terms of engagement,” and “human-first,” adding rich and nuanced insights to prior survey work on users’ views towards biometric-based emotional AI technologies. In contributing to a richer understanding of whether emotional AI technologies should be deployed in consumer-facing recommender systems, and if so, on what terms, we find that well-established policy-friendly criticisms apply to global emotional AI recommender systems. We conclude, however, that problems of alienation and need for a human-first approach to emerging AI technology are the most significant criticisms.

On manipulation by emotional AI: UK adults’ views and governance implications

“…Emotion profiling through biometrics includes techniques such as facial coding of expressions, voice analytics, eye-tracking, and wearables that sense skin responses, muscle activity, heart activity, respiration, and brain activity. Recent years have seen sector-specific trials of such techniques in security, education and workplaces across the world ( McStay, 2018 , 2020b , 2023 ; ARTICLE 19, 2021 ; Mantello et al, 2021 ; Urquhart et al, 2022 ); and biometric (voice-based) emotional AI in voice assistants is also envisaged in the patents of Amazon (the world’s largest online marketplace) to offer users highly tailored services, ads and products from the wider platform ( Bakir et al, 2023a ). Beyond trials and patents, use of biometrics to gauge emotions is also being rolled out in consumer-facing sectors such as in cars to improve cabin experience and safety ( McStay and Urquhart, 2022b ); in wearables to help users manage their mental health and day ( McStay, 2018 ); and in robotic toys to adapt and respond to users’ emotions, and to display the toy’s “moods” ( McStay and Rosner, 2021 ).…”

“…For instance, sociological literature (largely US-focused) highlights how racism (among other things) on data-driven discrimination shapes people’s experiences of data-driven systems ( Fisher, 2009 ; Madden et al, 2017 ; Benjamin, 2019 ). Regarding the group that self-identify as disabled, we reasoned that emotion profiling systems could have much to offer disabled people who may, for instance, be more reliant on technology to enable communication and employment, but that emotion profiling technologies can also risk prescribing value-laden benchmarks of what constitutes “normality” ( Bakir et al, 2023a ). We posited that separate focus groups based on ethnicity and disability might more readily surface unique insights on such factors that could be missed in more general focus groups.…”