Human involvement in autonomous decision-making systems. Lessons learned from three case studies in aviation, social care and road vehicles

Abstract

This paper draws on three case studies to examine some of the challenges and tensions involved in the use of Autonomous Decision-Making Systems (ADMS). In particular, the paper highlights: (i) challenges around the shifting “locale” of the decision, and the associated consequences for stakeholders; (ii) potential implications for stakeholders from regulation such as the General Data Protection Regulation (GDPR); (iii) the different values that stakeholder groups bring to the “decision” question; (iv) how complex pre-existing webs of stakeholders and decision-making authorities may be disrupted or disempowered by the use of an automated system and the lack of evaluation of possible consequences; (v) how ADMS for non-technical users can lead to circumvention of the boundaries of intended system use. We illustrate these challenges through case studies in three domains: adult social care, aviation, and vehicle driver monitoring systems. The paper closes with recommendations for both practice and policy in the deployment of ADMS.

Shared eHMI: Bridging Human–Machine Understanding in Autonomous Wheelchair Navigation

“…However, Human decision-making typically relies on a myriad of contextual factors, and many of these factors are extremely difficult to distill into discrete data streams and assign different weights. This level of nuanced decision-making is challenging for autonomous systems to fully replicate, and they are also susceptible to errors and biases, including false positives and omissions [55]. Consequently, if the system autonomously makes decisions, the ramifications of incorrect choices could lead to extremely serious consequences [56].…”