Against Algorithmic Exploitation of Human Vulnerabilities

Abstract

Decisions such as which movie to watch next, which song to listen to, or which product to buy online, are increasingly influenced by recommender systems and user models that incorporate information on users' past behaviours, preferences, and digitally created content. Machine learning models that enable recommendations and that are trained on user data may unintentionally leverage information on human characteristics that are considered vulnerabilities, such as depression, young age, or gambling addiction. The use of algorithmic decisions based on latent vulnerable state representations could be considered manipulative and could have a deteriorating impact on the condition of vulnerable individuals. In this paper, we are concerned with the problem of machine learning models inadvertently modelling vulnerabilities, and want to raise awareness for this issue to be considered in legislation and AI ethics. Hence, we define and describe common vulnerabilities, and illustrate cases where they are likely to play a role in algorithmic decision-making. We propose a set of requirements for methods to detect the potential for vulnerability modelling, detect whether vulnerable groups are treated differently by a model, and detect whether a model has created an internal representation of vulnerability. We conclude that explainable artificial intelligence methods may be necessary for detecting vulnerability exploitation by machine learning-based recommendation systems.

The Era of Artificial Intelligence Deception: Unraveling the Complexities of False Realities and Emerging Threats of Misinformation

“…The authors of [30] provided empirical evidence of AI's ability to manipulate human decision making by exploiting cognitive vulnerabilities through the adversarial framework. This work aligns with the broader discussions in the literature on the characterization of manipulation from AI systems [15], the exploitation of psychological vulnerabilities by machine learning models [16], and the role of cognitive biases in AI-assisted decision making. These findings underscore the importance of understanding and addressing the potential of AI systems to influence human behavior and the urgent need for ethical considerations in developing and deploying AI technologies.…”

“…The adversarial framework employed by [30] demonstrates how AI systems can learn manipulative behaviors from training data and unintentionally use manipulative tactics to optimize objectives. Moreover, the exploitation of cognitive vulnerabilities by AI systems, as shown in the experiments conducted by the authors, aligns with the discussion by [16] on how machine learning models can unintentionally leverage information on human vulnerabilities, such as depression, and how this can lead to manipulative or exploitative practices. The authors highlight how vulnerable individuals can be exposed to additional harm without their awareness, especially in digital environments where psychological vulnerabilities are more exposed to exploitation.…”

“…This integration prompts a reevaluation of the role of machines in shaping our choices, potentially overshadowing human judgment. AI's subtle yet pervasive influence affects individual decisions and has broader societal implications, as collective behaviors and norms may shift in response to algorithmic inputs [16].…”

“…The authors of [17] analyze AI-driven persuasion techniques, highlighting how AI's ability to engage users over extended periods, emulate authoritative personas, and tailor messages to user intentions can subtly guide decisions in powerful ways. The authors of [16] discussed how machine learning models can exploit human psychological vulnerabilities such as depression without user awareness, especially on digital platforms. This ties into the point made by [15] that the covertness of AI influence is a defining manipulation factor.…”

“…They advocate sociotechnical interventions, including the rigorous auditing and democratic oversight of AI systems, as essential tactics to preserve human autonomy and guarantee the ethical deployment and use of AI technologies. Fostering transparency, incorporating diverse stakeholder perspectives, and continuously monitoring systems' real-world impacts are vital approaches the literature suggests [16,17].…”