Multimodal Frame Identification with Multilingual Evaluation

Abstract

An essential step in FrameNet Semantic Role Labeling is the Frame Identification (FrameId) task, which aims at disambiguating a situation around a predicate. Whilst current FrameId methods rely on textual representations only, we hypothesize that FrameId can profit from a richer understanding of the situational context. Such contextual information can be obtained from common sense knowledge, which is more present in images than in text. In this paper, we extend a state-of-the-art FrameId system in order to effectively leverage multimodal representations. We conduct a comprehensive evaluation on the English FrameNet and its German counterpart SALSA. Our analysis shows that for the German data, textual representations are still competitive with multimodal ones. However on the English data, our multimodal FrameId approach outperforms its unimodal counterpart, setting a new state of the art. Its benefits are particularly apparent in dealing with ambiguous and rare instances, the main source of errors of current systems. For research purposes, we release (a) the implementation of our system, (b) our evaluation splits for SALSA 2.0, and (c) the embeddings for synsets and IMAGINED words.

Graph Embeddings for Frame Identification

“…Our model underperforms compared to other embedding frameworks from Hermann et al (2014) and Botschen et al (2018), which can be explained through an examination of the input representation methods used by the different models, as well as their disambiguation strategies. The model by Hermann et al (2014) constructs an input representation that encodes the syntactic dependency relations found within the predicate context by concatenating the embeddings for the arguments and learning a mapping to a lowerdimensional space.…”

“…The Botschen et al (2018) model is most significantly different from ours in two respects: it uses multimodal embedding representations at the input (textual + visual), and it employs a softmax classifier at the output step, whereas we use MSE as a loss function. Prior work has shown that the first option is more powerful in the context of word sense disambiguation tasks (Popov, 2017).…”

Frame- and Entity-Based Knowledge for Common-Sense Argumentative Reasoning

“…Work on event semantics hints at two annotation types complementing each other: additional information about participants benefits event prediction (Ahrendt and Demberg, 2016;Botschen et al, 2018) and context information about events benefits the prediction of implicit arguments and entities (Cheng and Erk, 2018). The complementarity is further affirmed by efforts on aligning WD and the FN lexicon: the best alignment approach only maps 37% of the total WD properties to frames (Mousselly-Sergieh and Gurevych, 2016).…”

“…Both annotation tools, the WD entity linker as well as the FN frame identifier, introduce some noise: for the entity linker, Sorokin and Gurevych (2018) report 0.73 F-score and the frame identifier has an accuracy of 0.89 (Botschen et al, 2018). We perform a manual error analysis on 50 instances of the test set to understand the effect of the noisy WD annotation.…”

“…We use two freely available systems to obtain semantic annotations for the claim (b), the reason (c) and the alternative warrants (i, ii): the frame identifier by Botschen et al (2018) for frame annotations and the entity linker by Sorokin and Gurevych (2018). We employ pre-trained vector representations to encode information from FN and WD.…”