Concept-based model explanations for electronic health records

Mincu, Diana; Loreaux, Eric; Hou, Shaobo; Baur, Sebastien; Protsyuk, Ivan; Seneviratne, Martin; Mottram, Anne; Tomašev, Nenad; Karthikesalingam, Alan; Schrouff, Jessica

doi:10.1145/3450439.3451858

Cited by 15 publications

(11 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Concept-based explanations have drawn much attention recently (Kim et al 2018;Chen et al 2019b;Goyal et al 2019;Wu et al 2020;Koh et al 2020;Yeh et al 2019b;Mincu et al 2021). Although these concept-based explanation methods are promising, their scalability is limited by the need for "humans-in-the-loop".…”

Section: Related Workmentioning

confidence: 99%

“…Although these concept-based explanation methods are promising, their scalability is limited by the need for "humans-in-the-loop". The methods proposed in Kim et al 2018;Chen et al 2019b;Mincu et al 2021) need human to manually define/extract concepts and quantify the importance score of each pre-defined concept in a post-hoc way. (Goyal et al 2019) directly performs the intervention of adding or removing a concept.…”

Section: Related Workmentioning

confidence: 99%

“…In contrast, human reasoning often comprises "concept-based thinking" by extracting similarities from numerous samples and grouping them semantically based on their resemblance (Yeh et al 2019b). As a consequence, recent research has focused on designing concept-based explanation methods to interpret how ML models use high-level human-understandable concepts in arriving at decisions (Kim et al 2018;Chen et al 2019b;Goyal et al 2019;Wu et al 2020;Koh et al 2020;Yeh et al 2019b;Mincu et al 2021).…”

Section: Introductionmentioning

confidence: 99%

“…The Thirty-Sixth AAAI Conference on Artificial Intelligence lenge here is how to define the units of concept-based explanations from the learning network structure considering that they are very subjective. If we directly follow existing concept-based works (Kim et al 2018;Chen et al 2019b;Goyal et al 2019;Wu et al 2020;Koh et al 2020;Yeh et al 2019b;Mincu et al 2021), the human has to be involved in this laborious tuning process. The reason is that ML models usually do not comprehend the way humans do and cannot guarantee that the extracted concepts are semantically meaningful, which also violates the fidelity of concept-based explanations.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Towards Automating Model Explanations with Certified Robustness Guarantees

Huai

Liu

Miao³

et al. 2022

AAAI

View full text Add to dashboard Cite

Providing model explanations has gained significant popularity recently. In contrast with the traditional feature-level model explanations, concept-based explanations can provide explanations in the form of high-level human concepts. However, existing concept-based explanation methods implicitly follow a two-step procedure that involves human intervention. Specifically, they first need the human to be involved to define (or extract) the high-level concepts, and then manually compute the importance scores of these identified concepts in a post-hoc way. This laborious process requires significant human effort and resource expenditure due to manual work, which hinders their large-scale deployability. In practice, it is challenging to automatically generate the concept-based explanations without human intervention due to the subjectivity of defining the units of concept-based interpretability. In addition, due to its data-driven nature, the interpretability itself is also potentially susceptible to malicious manipulations. Hence, our goal in this paper is to free human from this tedious process, while ensuring that the generated explanations are provably robust to adversarial perturbations. We propose a novel concept-based interpretation method, which can not only automatically provide the prototype-based concept explanations but also provide certified robustness guarantees for the generated prototype-based explanations. We also conduct extensive experiments on real-world datasets to verify the desirable properties of the proposed method.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Towards Automating Model Explanations with Certified Robustness Guarantees

Huai

Liu

Miao³

et al. 2022

AAAI

View full text Add to dashboard Cite

show abstract

“…Activations detecting move types from a square, illustrated with the activation input covariance cov(z 1i , z 0 ) for the first layer. From left to right: Activations i =(5,4,23) and(5,4,90) show a B and Q overlaid, and detect possible long-range diagonal piece movement from f3 and e4 (compare i =(5,4,15) in the bottom right of Figure10, which only detects the bishop). Activations i =(5,4,33) and(5,4,92) show a R and Q overlaid, and detect when the (opponent's) piece on e5 can move horizontally or vertically across the board.…”

mentioning

confidence: 99%

Acquisition of Chess Knowledge in AlphaZero

McGrath¹,

Kapishnikov²,

Tomašev³

et al. 2021

Preprint

View full text Add to dashboard Cite

What is learned by sophisticated neural network agents such as AlphaZero? This question is of both scientific and practical interest. If the representations of strong neural networks bear no resemblance to human concepts, our ability to understand faithful explanations of their decisions will be restricted, ultimately limiting what we can achieve with neural network interpretability. In this work we provide evidence that human knowledge is acquired by the AlphaZero neural network as it trains on the game of chess. By probing for a broad range of human chess concepts we show when and where these concepts are represented in the AlphaZero network. We also provide a behavioural analysis focusing on opening play, including qualitative analysis from chess Grandmaster Vladimir Kramnik. Finally, we carry out a preliminary investigation looking at the low-level details of AlphaZero's representations, and make the resulting behavioural and representational analyses available online.

show abstract