Multimodal Named Entity Recognition with Image Attributes and Image Knowledge

Chen, Fan; Li, Zhixu; Gu, Binbin; Chen, Zhigang

doi:10.1007/978-3-030-73197-7_12

Cited by 31 publications

(25 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We use the F1 score (F1) of each type and overall precision (P), recall (R) and F1 score (F1) to evaluate the performance of the MNER models, which are widely used in many recent works [2,12,14,19,22]. In this experiment, for a fair comparison, we use the code provided by [19] for evaluation 1 .…”

Section: Metricsmentioning

confidence: 99%

“…• MT-BERT-CRF [19], which is a variant of UMT-BERT-CRF without the auxiliary module. • ATTR-MMKG-MNER [2], which is a multimodal NER model that introduces both image attributes and image knowledge to help improve NER task. • MAF, which is the model we proposed in this paper.…”

Section: Baselinesmentioning

confidence: 99%

“…Table 2: Performance Comparison on Two MNER Dataset. For a fair comparison, we refer to the results of baselines before BERT-CRF and UMT-BERT-CRF with the marker ♣ from [19], and the result of ATTR-MMKG-MNER and T-NER with the marker ♠ from [2].…”

Section: Effectivenessmentioning

confidence: 99%

“…In this paper, we study MNER in social media posts. Compared with text-based NER methods, existing works have achieved good performance [2,12,14,19,22]. They mainly focus on using a crossmodal attention mechanism to combine text representation with image representation.…”

Section: Introductionmentioning

confidence: 99%

“…[19] proposes a multimodal transformer architecture, which captures the inter-modal interactions with a multimodal interaction module. In addition, [2] introduces image attributes and image knowledge to help capture the deep features of the image and improve the performance of the MNER model.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

MAF: A General Matching and Alignment Framework for Multimodal Named Entity Recognition

Huang

Sha

et al. 2022

Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining

View full text Add to dashboard Cite

In this paper, we study multimodal named entity recognition in social media posts. Existing works mainly focus on using a crossmodal attention mechanism to combine text representation with image representation. However, they still suffer from two weaknesses: (1) the current methods are based on a strong assumption that each text and its accompanying image are matched, and the image can be used to help identify named entities in the text. However, this assumption is not always true in real scenarios, and the strong assumption may reduce the recognition effect of the MNER model;(2) the current methods fail to construct a consistent representation to bridge the semantic gap between two modalities, which prevents the model from establishing a good connection between the text and image. To address these issues, we propose a general matching and alignment framework (MAF) for multimodal named entity recognition in social media posts. Specifically, to solve the first issue, we propose a novel cross-modal matching (CM) module to calculate the similarity score between text and image, and use the score to determine the proportion of visual information that should be retained. To solve the second issue, we propose a novel cross-modal alignment (CA) module to make the representations of the two modalities more consistent. We conduct extensive experiments, ablation studies, and case studies to demonstrate the effectiveness and efficiency of our method.The source code of this paper can be found in https://github.com/xubodhu/MAF.

show abstract