Deep learning, transparency, and trust in human robot teamwork

Lewis, Michael; Li, Huao; Sycara, Katia

doi:10.1016/b978-0-12-819472-0.00014-9

Cited by 20 publications

(19 citation statements)

References 76 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Item three relates to the perceived inference of the agent’s intentions by the human. It is one of the variables focused on the transparency of the agent, which plays a key role in constructing human trust in an agent ( Lewis et al, 2021 ). From the concrete algorithm perspective, a higher score is expected for guidance agents (especially explicit guidance agents) than for supportive agents.…”

Section: Methodsmentioning

confidence: 99%

Balancing Performance and Human Autonomy With Implicit Guidance Agent

Nakahashi

Yamada

2021

Front. Artif. Intell.

View full text Add to dashboard Cite

The human-agent team, which is a problem in which humans and autonomous agents collaborate to achieve one task, is typical in human-AI collaboration. For effective collaboration, humans want to have an effective plan, but in realistic situations, they might have difficulty calculating the best plan due to cognitive limitations. In this case, guidance from an agent that has many computational resources may be useful. However, if an agent guides the human behavior explicitly, the human may feel that they have lost autonomy and are being controlled by the agent. We therefore investigated implicit guidance offered by means of an agent’s behavior. With this type of guidance, the agent acts in a way that makes it easy for the human to find an effective plan for a collaborative task, and the human can then improve the plan. Since the human improves their plan voluntarily, he or she maintains autonomy. We modeled a collaborative agent with implicit guidance by integrating the Bayesian Theory of Mind into existing collaborative-planning algorithms and demonstrated through a behavioral experiment that implicit guidance is effective for enabling humans to maintain a balance between improving their plans and retaining autonomy.

show abstract

Section: Methodsmentioning

confidence: 99%

Balancing Performance and Human Autonomy With Implicit Guidance Agent

Nakahashi

Yamada

2021

Front. Artif. Intell.

View full text Add to dashboard Cite

show abstract

“…This opacity indeed engendered uncertainty, disturbing the process tasks accomplishment for which the expert system was involved [23,22]. The problems encountered during the expert system's design are even more topical for contemporary AI-powered solutions as new ML solutions, such as deep neural networks, are less understandable and more opaque than IF-THEN rules [14,24]. In addition, AI-based solutions are being implemented in safety-critical domains, such as healthcare and aviation 1 [25,26].…”

Section: Why Explain?mentioning

confidence: 99%

“…Explanations can be visual, textual, or more complex explanations, and the type of explanations depends on the users and its context. While some users require partial explanations focusing on relevant features, others may require the complete portrait of features and reasoning behind the decision [49,24]. Moreover, some critical systems might also be required to provide specific explanations to official regulatory agencies, so their mandates must also be fulfilled.…”

Section: What and How To Explain?mentioning

confidence: 99%

See 1 more Smart Citation

Towards an Explanation Space to Align Humans and Explainable-AI Teamwork

Cabour,

Morales,

Ledoux

et al. 2021

Preprint

View full text Add to dashboard Cite

Providing meaningful and actionable explanations to end-users is a fundamental prerequisite for implementing explainable intelligent systems in the real world. Explainability is a situated interaction between a user and the AI system rather than being static design principles. The content of explanations is context-dependent and must be defined by evidence about the user and its context. This paper seeks to operationalize this concept by proposing a formative architecture that defines the explanation space from a user-inspired perspective. The architecture comprises five intertwined components to outline explanation requirements for a task: (1) the end-users mental models, (2) the end-users cognitive process, (3) the user interface, (4) the human-explainer agent, and the (5) agent process. We first define each component of the architecture. Then we present the Abstracted Explanation Space, a modeling tool that aggregates the architecture's components to support designers in systematically aligning explanations with the end-users work practices, needs, and goals. It guides the specifications of what needs to be explained (content -end-users mental model), why this explanation is necessary (context -end-users cognitive process), to delimit how to explain it (format -human-explainer agent and user interface), and when should the explanations be given. We then exemplify the tool's use in an ongoing case study in the aircraft maintenance domain. Finally, we discuss possible contributions of the tool, known limitations/areas for improvement, and future work to be done.

show abstract

“…Item 3 relates to the perceived inference of the agent's intentions by the human. It is one of the variables focused on the transparency of the agent, which plays a key role in constructing human trust in an agent [Lewis et al, 2021]. From the concrete algorithm perspective, a higher score is expected for guidance agents (especially explicit guidance agents) than for supportive agents.…”

Section: Figure 3: Example Of Experimentsmentioning

confidence: 99%

Balancing Performance and Human Autonomy with Implicit Guidance Agent

Nakahashi¹,

Yamada²

2021

Preprint

View full text Add to dashboard Cite

The human-agent team, which is a problem in which humans and autonomous agents collaborate to achieve one task, is typical in human-AI collaboration. For effective collaboration, humans want to have an effective plan, but in realistic situations, they might have difficulty calculating the best plan due to cognitive limitations. In this case, guidance from an agent that has many computational resources may be useful. However, if an agent guides the human behavior explicitly, the human may feel that they have lost autonomy and are being controlled by the agent. We therefore investigated implicit guidance offered by means of an agent's behavior. With this type of guidance, the agent acts in a way that makes it easy for the human to find an effective plan for a collaborative task, and the human can then improve the plan. Since the human improves their plan voluntarily, he or she maintains autonomy. We modeled a collaborative agent with implicit guidance by integrating the Bayesian Theory of Mind into existing collaborative-planning algorithms and demonstrated through a behavioral experiment that implicit guidance is effective for enabling humans to maintain a balance between improving their plans and retaining autonomy.

show abstract

Deep learning, transparency, and trust in human robot teamwork

Cited by 20 publications

References 76 publications

Balancing Performance and Human Autonomy With Implicit Guidance Agent

Balancing Performance and Human Autonomy With Implicit Guidance Agent

Towards an Explanation Space to Align Humans and Explainable-AI Teamwork

Balancing Performance and Human Autonomy with Implicit Guidance Agent

Contact Info

Product

Resources

About