Aiding Intra-Text Representations with Visual Context for Multimodal Named Entity Recognition

Abstract

With massive explosion of social media such as Twitter and Instagram, people daily share billions of multimedia posts, containing images and text. Typically, text in these posts is short, informal and noisy, leading to ambiguities which can be resolved using images. In this paper we explore text-centric Named Entity Recognition task on these multimedia posts. We propose an end to end model which learns a joint representation of a text and an image. Our model extends multi-dimensional self attention technique, where now image help to enhance relationship between words. Experiments show that our model is capable of capturing both textual and visual contexts with greater accuracy, achieving state-of-the-art results on Twitter multimodal Named Entity Recognition dataset.• Unrelated image : Text information do not match with an image, as we can see in Fig. 8(a), "Reddit" belongs to

RIVA: A Pre-trained Tweet Multimodal Model Based on Text-image Relation for Multimodal NER

“…The output score of the filtration gate was computed by a sigmoid activation function. Arshad et al (2019) also presented a gated multimodal fusion representation for each token. The gated fusion is a weighted sum of visual attention feature and token alignment feature.…”

“…We illustrate four failure examples mentioned in (Lu et al, 2018) and(Arshad et al, 2019) in Table 7.…”

“…One of the reasons is that tweets are short messages and the context for inference is insufficient. Recent works on tweets based on multimodal learning have been increasing (Moon et al, 2018;Lu et al, 2018;Arshad et al, 2019). The researchers attempted to improve the performance of NER in tweets with the aid of visual clues.…”

“…Most of the multimodal NER (MNER) methods used attention weights to extract visual clues related to the NEs (Lu et al, 2018;Arshad et al, 2019) . The visual attention-based models always assume that the images in tweets are related to the texts, such as words in the text are represented in the image, e.g., Figure 1(a) shows a successful visual attention example from Lu et al (2018).…”

Picture What You Read

“…Typically, users combine text, image, audio or video to sell a product over an e-commence platform or express views on social media. The combination of these media types has been extensively studied to solve various tasks including classification [1], [2], [3], cross-modal retrieval [4] semantic relatedness [5], [6], image captioning [7], [8], multimodal named entity recognition [9], [10] and Visual Question Answering [11], [12]. In addition, multimodal data fueled an increased interest in generating images conditioned on natural language [13], [14].…”