Building Explainable Artificial Intelligence Systems

Core, Mark G.; Lane, H. Chad; Lent, Michael van; Gomboc, Dave; Solomon, Steve; Rosenberg, Milton

doi:10.21236/ada459166

Cited by 90 publications

(54 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Early textual explanation models are applied to medical images [25] and developed as a feedback for teaching programs [12,27,3]. These systems are mainly template based.…”

Section: Related Workmentioning

confidence: 99%

Grounding Visual Explanations

Hendricks

Darrell

et al. 2018

Lecture Notes in Computer Science

167

121

View full text Add to dashboard Cite

Existing visual explanation generating agents learn to fluently justify a class prediction. However, they may mention visual attributes which reflect a strong class prior, although the evidence may not actually be in the image. This is particularly concerning as ultimately such agents fail in building trust with human users. To overcome this limitation, we propose a phrase-critic model to refine generated candidate explanations augmented with flipped phrases which we use as negative examples while training. At inference time, our phrase-critic model takes an image and a candidate explanation as input and outputs a score indicating how well the candidate explanation is grounded in the image. Our explainable AI agent is capable of providing counter arguments for an alternative prediction, i.e. counterfactuals, along with explanations that justify the correct classification decisions. Our model improves the textual explanation quality of fine-grained classification decisions on the CUB dataset by mentioning phrases that are grounded in the image. Moreover, on the FOIL tasks, our agent detects when there is a mistake in the sentence, grounds the incorrect phrase and corrects it significantly better than other models.

show abstract

“…Early textual explanation models are applied to medical images [25] and developed as a feedback for teaching programs [12,27,3]. These systems are mainly template based.…”

Section: Related Workmentioning

confidence: 99%

Grounding Visual Explanations

Hendricks

Darrell

et al. 2018

Lecture Notes in Computer Science

167

121

View full text Add to dashboard Cite

show abstract

“…Broadly, this principle is in line with the considerable body of research in providing explanations in intelligent systems. (See, for example, the earliest expert systems that explained their reasoning in terms of the chain of rules used [Clancey 1983] to more recent work in conjunction with recommenders [Glass et al 2008], architectures for explainable AI [Core et al 2006], and the engineering of service-oriented adaptation [Koidl and Conlan 2008].) That work confirms that some representation and reasoning approaches are easier for people to understand.…”

Section: Principles For Creating Scrutable User Modelsmentioning

confidence: 77%

Creating personalized systems that people can scrutinize and control

Kay

Kummerfeld

2012

ACM Trans. Interact. Intell. Syst.

View full text Add to dashboard Cite

Widespread personalized computing systems play an already important and fast-growing role in diverse contexts, such as location-based services, recommenders, commercial Web-based services, and teaching systems. The personalization in these systems is driven by information about the user, a user model. Moreover, as computers become both ubiquitous and pervasive, personalization operates across the many devices and information stores that constitute the user's personal digital ecosystem. This enables personalization, and the user models driving it, to play an increasing role in people's everyday lives. This makes it critical to establish ways to address key problems of personalization related to privacy, invisibility of personalization, errors in user models, wasted user models, and the broad issue of enabling people to control their user models and associated personalization. We offer scrutable user models as a foundation for tackling these problems.This article argues the importance of scrutable user modeling and personalization, illustrating key elements in case studies from our work. We then identify the broad roles for scrutable user models. The article describes how to tackle the technical and interface challenges of designing and building scrutable user modeling systems, presenting design principles and showing how they were established over our twenty years of work on the Personis software framework. Our contributions are the set of principles for scrutable personalization linked to our experience from creating and evaluating frameworks and associated applications built upon them. These constitute a general approach to tackling problems of personalization by enabling users to scrutinize their user models as a basis for understanding and controlling personalization.

show abstract

“…Hence, while finding an optimal deceptive policy, or enumerating all optimal deceptive policies, may be sufficient in order to deceive an adversary, analyzing and learning from deceptive behavior in one scenario in order to determine deceptive behavior in a similar scenario would require us to describe the set of deceptive policies in understandable terms. Such a question broadly falls within the research effort on explainable artificial intelligence [51]- [53].…”

Section: Discussionmentioning

confidence: 99%

Deception in Optimal Control

Ornik

Topcu

2018

2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton)

View full text Add to dashboard Cite

In this paper, we consider an adversarial scenario where one agent seeks to achieve an objective and its adversary seeks to learn the agent's intentions and prevent the agent from achieving its objective. The agent has an incentive to try to deceive the adversary about its intentions, while at the same time working to achieve its objective. The primary contribution of this paper is to introduce a mathematically rigorous framework for the notion of deception within the context of optimal control. The central notion introduced in the paper is that of a belief-induced reward: a reward dependent not only on the agent's state and action, but also adversary's beliefs. Design of an optimal deceptive strategy then becomes a question of optimal control design on the product of the agent's state space and the adversary's belief space. The proposed framework allows for deception to be defined in an arbitrary control system endowed with a reward function, as well as with additional specifications limiting the agent's control policy. In addition to defining deception, we discuss design of optimally deceptive strategies under uncertainties in agent's knowledge about the adversary's learning process. In the latter part of the paper, we focus on a setting where the agent's behavior is governed by a Markov decision process, and show that the design of optimally deceptive strategies under lack of knowledge about the adversary naturally reduces to previously discussed problems in control design on partially observable or uncertain Markov decision processes. Finally, we present two examples of deceptive strategies: a "cops and robbers" scenario and an example where an agent may use camouflage while moving. We show that optimally deceptive strategies in such examples follow the intuitive idea of how to deceive an adversary in the above settings.

show abstract

Building Explainable Artificial Intelligence Systems

Cited by 90 publications

References 5 publications

Grounding Visual Explanations

Grounding Visual Explanations

Creating personalized systems that people can scrutinize and control

Deception in Optimal Control

Contact Info

Product

Resources

About