Pathologies of Neural Models Make Interpretations Difficult

Feng, Shi; Wallace, Eric; Grissom, Alvin; Iyyer, Mohit; Rodríguez, Pedro; Boyd-Graber, Jordan

doi:10.18653/v1/d18-1407

Cited by 175 publications

(98 citation statements)

References 25 publications

(34 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We consider the Slug system , a seq2seq-based ensemble system, as the overall winner of this challenge. It received high human ratings for both 30 Note that this problem appears to be more general since it has also been reported in related fields, including image captioning Rohrbach et al (2018), machine translation Koehn and Knowles (2017); Lee et al (2019), and question answering Feng et al (2018). naturalness and quality, as well as for automatic word-overlap metrics.…”

Section: Winning Systemmentioning

confidence: 74%

Evaluating the state-of-the-art of End-to-End Natural Language Generation: The E2E NLG challenge

Dušek

Novikova

Rieser

2020

Computer Speech & Language

170

183

View full text Add to dashboard Cite

This paper provides a comprehensive analysis of the first shared task on End-to-End Natural Language Generation (NLG) and identifies avenues for future research based on the results. This shared task aimed to assess whether recent end-to-end NLG systems can generate more complex output by learning from datasets containing higher lexical richness, syntactic complexity and diverse discourse phenomena. Introducing novel automatic and human metrics, we compare 62 systems submitted by 17 institutions, covering a wide range of approaches, including machine learning architectures -with the majority implementing sequence-to-sequence models (seq2seq) -as well as systems based on grammatical rules and templates. Seq2seq-based systems have demonstrated a great potential for NLG in the challenge. We find that seq2seq systems generally score high in terms of word-overlap metrics and human evaluations of naturalness -with the winning Slug system (Juraska et al., 2018) being seq2seq-based.However, vanilla seq2seq models often fail to correctly express a given meaning representation if they lack a strong semantic control mechanism applied during decoding.Moreover, seq2seq models can be outperformed by hand-engineered systems in terms of overall quality, as well as complexity, length and diversity of outputs. This research has influenced, inspired and motivated a number of recent studies outwith the original competition, which we also summarise as part of this paper.

show abstract

Section: Winning Systemmentioning

confidence: 74%

Evaluating the state-of-the-art of End-to-End Natural Language Generation: The E2E NLG challenge

Dušek

Novikova

Rieser

2020

Computer Speech & Language

170

183

View full text Add to dashboard Cite

show abstract

“…Feng et al [ 159 ] introduced a process, called “input reduction”, which can expose issues regarding overconfidence and oversensitivity in natural language processing models. Under input reduction, non-important words are removed from the input text in interative fashion, while the model’s prediction for that input remains unchanged.…”

Section: Different Scopes Of Machine Learning Interpretability: a mentioning

confidence: 99%

Explainable AI: A Review of Machine Learning Interpretability Methods

Linardatos

Papastefanopoulos

Kotsiantis

2020

Entropy

1,429

697

View full text Add to dashboard Cite

Recent advances in artificial intelligence (AI) have led to its widespread industrial adoption, with machine learning systems demonstrating superhuman performance in a significant number of tasks. However, this surge in performance, has often been achieved through increased model complexity, turning such systems into “black box” approaches and causing uncertainty regarding the way they operate and, ultimately, the way that they come to decisions. This ambiguity has made it problematic for machine learning systems to be adopted in sensitive yet critical domains, where their value could be immense, such as healthcare. As a result, scientific interest in the field of Explainable Artificial Intelligence (XAI), a field that is concerned with the development of new methods that explain and interpret machine learning models, has been tremendously reignited over recent years. This study focuses on machine learning interpretability methods; more specifically, a literature review and taxonomy of these methods are presented, as well as links to their programming implementations, in the hope that this survey would serve as a reference point for both theorists and practitioners.

show abstract

“…One shortcoming of neural machine translation (NMT), and neural models in general, is that it is often difficult for humans to comprehend the reasons why the model is making predictions (Feng et al, 2018;Ghorbani et al, 2019). The main cause of such a difficulty is that in neural models, information is implicitly represented by realvalued vectors, and conceptual interpretation of these vectors remains a challenge.…”

Section: Introductionmentioning

confidence: 99%

Interrogating the Explanatory Power of Attention in Neural Machine Translation

Moradi¹,

Kambhatla²,

Sarkar³

2019

Proceedings of the 3rd Workshop on Neural Generation and Translation

View full text Add to dashboard Cite

Attention models have become a crucial component in neural machine translation (NMT). They are often implicitly or explicitly used to justify the model's decision in generating a specific token but it has not yet been rigorously established to what extent attention is a reliable source of information in NMT. To evaluate the explanatory power of attention for NMT, we examine the possibility of yielding the same prediction but with counterfactual attention models that modify crucial aspects of the trained attention model. Using these counterfactual attention mechanisms we assess the extent to which they still preserve the generation of function and content words in the translation process. Compared to a state of the art attention model, our counterfactual attention models produce 68% of function words and 21% of content words in our German-English dataset. Our experiments demonstrate that attention models by themselves cannot reliably explain the decisions made by a NMT model.

show abstract

Pathologies of Neural Models Make Interpretations Difficult

Abstract: arXiv:1804.07781v3 [cs.CL]

Cited by 175 publications

References 25 publications

Evaluating the state-of-the-art of End-to-End Natural Language Generation: The E2E NLG challenge

Evaluating the state-of-the-art of End-to-End Natural Language Generation: The E2E NLG challenge

Explainable AI: A Review of Machine Learning Interpretability Methods

Interrogating the Explanatory Power of Attention in Neural Machine Translation

Contact Info

Product

Resources

About