Multimodal Named Entity Recognition for Short Social Media Posts

Abstract

We introduce a new task called Multimodal Named Entity Recognition (MNER) for noisy user-generated data such as tweets or Snapchat captions, which comprise short text with accompanying images. These social media posts often come in inconsistent or incomplete syntax and lexical notations with very limited surrounding textual contexts, bringing significant challenges for NER. To this end, we create a new dataset for MNER called SnapCaptions (Snapchat image-caption pairs submitted to public and crowd-sourced stories with fully annotated named entities). We then build upon the state-of-the-art Bi-LSTM word/character based NER models with 1) a deep image network which incorporates relevant visual context to augment textual information, and 2) a generic modality-attention module which learns to attenuate irrelevant modalities while amplifying the most informative ones to extract contexts from, adaptive to each sample and token. The proposed MNER model with modality attention significantly outperforms the state-of-the-art text-only NER models by successfully leveraging provided visual contexts, opening up potential applications of MNER on myriads of social media platforms.

Aiding Intra-Text Representations with Visual Context for Multimodal Named Entity Recognition

“…On the other hand, textual attention techniques find semantic or syntactic alignments in handling long-term dependencies. Attention techniques have been extensively employed to vision and text related tasks, such as Image Captioning [14], VQA [9], Cross-Modal Retrieval [15] NER [10], [11], [12]. Multimodal Named Entity Recognition.…”

Multimodal Named Entity Disambiguation for Noisy Social Media Posts

“…f (•) is a transformation function with stacked layers that projects weighted input to the KB embeddings space,ỹ refers to the embeddings of negative samples randomly sampled from KB entities except the ground truth label of the instance, W = {W f ,W c ,W w ,W v } are the learnable parameters for f , c, w, and v respectively, and R(W) is a weight decay regularization term. Similarly to (Moon et al, 2018), we formulate the modality attention module for our MNED network as follows, which selectively attenuates or amplifies modalities:…”

“…Multimodal learning studies learning of a joint model that leverages contextual information from multiple modalities in parallel. Some of the relevant multimodal learning task to our MNED system include the multimodal named entity recognition task (Moon et al, 2018), which leverages both text and image to classify each token in a sentence to named entity or not. In their work, they employ an entity LSTM that takes as input each modality, and a softmax layer that outputs an entity label at each decoding step.…”

Fusion Strategy for Prosodic and Lexical Representations of Word Importance

“…We compared against models based on different multimodal feature representation strategies: Concatenation (CONCAT) [35,11,36]: The model creates a multimodal representation of words by simply concatenating the unimodal features at the word level. Attention-based Weighted Sum (ATTN) [37,16]: Instead of concatenating the unimodal signals as alternative feature vectors, the model uses an attention network to decide how to combine the information for the final representation. Tensor Fusion Network (TFN) [14]: This strategy models both the modality-specific and cross-modal interactions by computing an outer product over a set of unimodal vectors (with an extra constant dimension 1) rather than just the concatenation.…”