Does External Knowledge Help Explainable Natural Language Inference? Automatic Evaluation vs. Human Ratings

Abstract

Natural language inference (NLI) requires models to learn and apply commonsense knowledge. These reasoning abilities are particularly important for explainable NLI systems that generate a natural language explanation in addition to their label prediction. The integration of external knowledge has been shown to improve NLI systems, here we investigate whether it can also improve their explanation capabilities. For this, we investigate different sources of external knowledge and evaluate the performance of our models on in-domain data as well as on special transfer datasets that are designed to assess finegrained reasoning capabilities. We find that different sources of knowledge have a different effect on reasoning abilities, for example, implicit knowledge stored in language models can hinder reasoning on numbers and negations. Finally, we conduct the largest and most fine-grained explainable NLI crowdsourcing study to date. It reveals that even large differences in automatic performance scores do neither reflect in human ratings of label, explanation, commonsense nor grammar correctness.

Towards the Generation of Musical Explanations with GPT-3

“…WordNet, ConceptNet, etc.) in order to improve the explainability of Natural Language Inference (NLI) models [22]. Here we follow a similar approach, carrying out an initial study on the use of language models, particularly GPT-3, as a external source to generate explanations of musical decisions.…”

How to do human evaluation: A brief introduction to user studies in NLP

“…Both candidates receive identical BLEU-2 scores; however from a human perspective, sentence (a) seems to much better reflect the original German sentence. a Similarly, automatic evaluation measures used by other NLP tasks face the same problem (Callison-Burch et al 2006;Liu et al 2016;Mathur et al 2020;Schuff et al 2020Schuff et al , 2021Iskender et al 2020;Clinciu et al 2021). Therefore, human evaluation has begun to gain more and more attention in the NLP community (especially in the context of natural language generation tasks, including machine translation Belz and Reiter 2006;Novikova, Dusek, and Rieser 2018;van der Lee et al 2019).…”