Design and Evaluation of Metaphor Processing Systems

Abstract

System design and evaluation methodologies receive significant attention in natural language processing (NLP), with the systems typically being evaluated on a common task and against shared data sets. This enables direct system comparison and facilitates progress in the field. However, computational work on metaphor is considerably more fragmented than similar research efforts in other areas of NLP and semantics. Recent years have seen a growing interest in computational modeling of metaphor, with many new statistical techniques opening routes for improving system accuracy and robustness. However, the lack of a common task definition, shared data set, and evaluation strategy makes the methods hard to compare, and thus hampers our progress as a community in this area. The goal of this article is to review the system features and evaluation strategies that have been proposed for the metaphor processing task, and to analyze their benefits and downsides, with the aim of identifying the desired properties of metaphor processing systems and a set of requirements for their evaluation.

End-to-End Sequential Metaphor Identification Inspired by Linguistic Theories

“…Metaphor identification is a linguistic metaphor processing task that identifies metaphors in textual data, which is different from conceptual metaphor processing that maps concepts between source and target domains (Shutova, 2016), based on Conceptual Metaphor Theory (Lakoff and Johnson, 1980). In linguistic metaphor processing a metaphor is identified when the contextual meaning of a word contrasts with its literal meaning (summarised as MIP by Group (2007) and Steen et al (2010)).…”

Using Automated Metaphor Identification to Aid in Detection and Prediction of First-Episode Schizophrenia

“…Metaphor detection methods differ in how they define the task of metaphor detection-for instance, some algorithms seek to determine whether a phrase (such as sweet victory) is metaphorical (Krishnakumaran and Zhu, 2007;Turney et al, 2011;Tsvetkov et al, 2014;Bracewell et al, 2014;Gutiérrez et al, 2016), while others attempt to tag metaphoricity at the level of the utterance (Dunn, 2013a), or at the level of individual tokens in running text (Klebanov et al;Schulder and Hovy, 2014;Do Dinh and Gurevych, 2016). For a recent review, see Shutova (2015). For our purposes, we decided that tokenlevel metaphor detection offered the most appropriate level of granularity, and we chose the algorithm of (Do Dinh and Gurevych, 2016) because of its state-of-the-art performance at this task at the time we began this project.…”

Weeding out Conventionalized Metaphors: A Corpus of Novel Metaphor Annotations

“…Metaphors have received considerable interest in NLP in recent years (see Shutova (2015)). Research questions range from direct detection of metaphors in text (linguistic metaphors) to finding mappings between conceptual source and target domains (conceptual metaphors).…”