Human Judgement in Algorithmic Loops; Individual Justice and Automated Decision-Making

Abstract

There are various arguments in favour of tempering algorithmic decision-making with human judgement. One common family of arguments appeal to concepts and criteria derived from legal philosophy about the nature of law and legal reasoning, and argue that algorithmic systems cannot satisfy them (but humans can). This paper argues that among the latter family of arguments, there is often an implicit appeal to the notion that each case needs to be assessed on its own merits, without comparison to or generalisation from previous cases. This notion of 'individual justice' has featured in jurisprudential debates about the granularity of rules and tests, and the (in)justice of discrimination, but has not yet been explicitly imported into debates about justice and algorithmic decision-making. This paper has several aims. The first is to provide an overview of the concept of individual justice and distinguish it from related but distinct arguments about the value of human discretion. Equipped with this account of human judgement as a guarantor of individual justice, the second aim is to consider its place within and beside algorithmic decision-making. It argues that in so far as individual justice is valuable, it can only be meaningfully served through human judgement, because it antithetical to the kind of predetermined reasoning that characterises algorithmic systems. This suggests that-to the extent that individual justice is deemed important-a requirement for human intervention and oversight over algorithmic decisions is necessary. The third aim is to consider how individual justice relates to other dimensions of justice, namely consistency and fairness or non-discrimination. Finally, the article discusses two challenges that are raised by this account. The first challenge concerns how individual justice can be accommodated alongside other dimensions of justice in the socio-technical contexts in Human Judgement in Algorithmic Loops DRAFT 2 which humans-in-the-loop are situated. The second concerns the potential inequities in individual justice that might result from an uneven application of human judgement in algorithmic settings.

The agency of algorithms: Understanding human-algorithm interaction in administrative decision-making

“…Assessing whether algorithms actually deliver on these points goes beyond the scope of this article. Instead, the focus here is on the concerns voiced in the literature regarding the compatibility of algorithmic decision-making with principles of just and fair administration (Binns, 2019;Simmons, 2019). For example, studies in policing and criminal justice have argued that algorithmic decisions tend to reproduce bias towards already over-policed areas and target groups (Hannah-Moffat, 2016;Harcourt, 2007;Van Eijk, 2017).…”

“…A second major concern is the reduction of the discretionary space for human decision-makers to override algorithmic decisions (Citron & Pasquale, 2014;Zouridis et al, 2020). 'Keeping humans in the loop' (Zarsky, 2011) is seen as crucial to assure individual administrative justice, to override potential errors and to adapt decisions to specific circumstances, as is also common practice in classic 'analogue' decision-making through regulated street-level discretion (Binns 2019;Van Eck, 2018).…”

Artificial intelligence, bureaucratic form, and discretion in public service

“…Much of this work has focused on AI's potential impact on the economy, the labor market, and national defense (Frank et al, 2019;Frey & Osborne, 2017;Korinek & Stiglitz, 2017;McClure, 2018). Another vein of related work focuses on the use of machine learning-based decision support systems in criminal justice for bail setting and sentencing (Binns, 2019;Hannah-Moffat, 2013Hannah-Moffat et al, 2009). Less attention, however, has been paid to AI's implications for public organizations and the governance of the public sector more broadly.…”

Algorithmic Long-Term Unemployment Risk Assessment in Use: Counselors’ Perceptions and Use Practices

“…The frontline workers' exercise of discretion, or how they make decisions within their job constraints, has been captivating scholarly interest for decades (e.g., Lipsky 2010). The introduction of algorithms at work only added to the complexity of the workers' tasks and the importance of this topic; these workers are simultaneously managing numerous policies and programs, trying to live up to social expectations, and coping with the constraints imposed by increasingly complex digital tools (Binns 2019;Hupe 2019;Raso 2017).…”