Exploiting BERT for Multimodal Target Sentiment Classification through Input Space Translation

Khan, Zaid; Fu, Yun

doi:10.1145/3474085.3475692

Cited by 67 publications

(35 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since contents in different modalities are often closely related, exploiting such multimodal information can help better analyze users' sentiments towards different aspects. Recent studies on multimodal ABSA mainly concentrate on simple ABSA tasks such as multimodal ATE [182,183] and multimodal ASC [184,185,186,187]. To align the information from different modalities, the text and image are often first encoded to feature representations, then some interaction networks are designed to fuse the information for making the final prediction.…”

Section: Multimodal Absamentioning

confidence: 99%

A Survey on Aspect-Based Sentiment Analysis: Tasks, Methods, and Challenges

Zhang¹,

Li²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

As an important fine-grained sentiment analysis problem, aspect-based sentiment analysis (ABSA), aiming to analyze and understand people's opinions at the aspect level, has been attracting considerable interest in the last decade. To handle ABSA in different scenarios, various tasks have been introduced for analyzing different sentiment elements and their relations, including the aspect term, aspect category, opinion term, and sentiment polarity. Unlike early ABSA works focusing on a single sentiment element, many compound ABSA tasks involving multiple elements have been studied in recent years for capturing more complete aspect-level sentiment information. However, a systematic review of various ABSA tasks and their corresponding solutions is still lacking, which we aim to fill in this survey. More specifically, we provide a new taxonomy for ABSA which organizes existing studies from the axes of concerned sentiment elements, with an emphasis on recent advances of compound ABSA tasks. From the perspective of solutions, we summarize the utilization of pre-trained language models for ABSA, which improved the performance of ABSA to a new stage. Besides, techniques for building more practical ABSA systems in cross-domain/lingual scenarios are discussed. Finally, we review some emerging topics and discuss some open challenges to outlook potential future directions of ABSA.

show abstract

Section: Multimodal Absamentioning

confidence: 99%

A Survey on Aspect-Based Sentiment Analysis: Tasks, Methods, and Challenges

Zhang¹,

Li²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Multimodal Aspect-Based Sentiment Analysis. As an important sentiment analysis task, many approaches have been approached to tackle the three subtasks of MABSA, including Multimodal Aspect Term Extraction Wu et al, 2020a,b;Sun et al, 2020;, Multimodal Aspect Sentiment Classification (Xu et al, 2019;Yu et al, 2020a;Yang et al, 2021a;Khan and Fu, 2021) and Joint Multimodal Aspect-Sentiment Analysis (Ju et al, 2021). In this work, we aim to propose a general pre-training framework to improve the performance of all the three subtasks.…”

Section: Related Workmentioning

confidence: 99%

“…1) TomBERT (Yu and Jiang, 2019), which tackles the MASC task by employing BERT to capture intra-modality dynamics. 2) CapTrBERT (Khan and Fu, 2021), which translates the image to a caption as an auxiliary sentence for sentiment classification.…”

Section: Compared Systemsmentioning

confidence: 99%

“…Previous research mostly focused on its two subtasks, including Multimodal Aspect Term Extraction (MATE) and Multimodal Aspect-oriented Sentiment Classification (MASC). Given a text-image pair as input, MATE aims to extract all the aspect terms mentioned in the text Lu et al, 2018;Wu et al, 2020a,b; aims to classify the sentiment towards each extracted aspect term (Xu et al, 2019;Yu and Jiang, 2019;Khan and Fu, 2021). As the two subtasks are closely related to each other, Ju et al (2021) recently introduced the Joint Multimodal Aspect-Sentiment Analysis (JMASA) task, aiming to jointly extract the aspect terms and their corresponding sentiments.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis

Liu¹,

Yu²,

Xia³

2022

Preprint

View full text Add to dashboard Cite

As an important task in sentiment analysis, Multimodal Aspect-Based Sentiment Analysis (MABSA) has attracted increasing attention in recent years. However, previous approaches either (i) use separately pre-trained visual and textual models, which ignore the crossmodal alignment or (ii) use vision-language models pre-trained with general pre-training tasks, which are inadequate to identify finegrained aspects, opinions, and their alignments across modalities. To tackle these limitations, we propose a task-specific Vision-Language Pre-training framework for MABSA (VLP-MABSA), which is a unified multimodal encoder-decoder architecture for all the pretraining and downstream tasks. We further design three types of task-specific pre-training tasks from the language, vision, and multimodal modalities, respectively. Experimental results show that our approach generally outperforms the state-of-the-art approaches on three MABSA subtasks. Further analysis demonstrates the effectiveness of each pretraining task. The source code is publicly released at https://github.com/NUSTM/ VLP-MABSA.

show abstract

“…Over the past decades, statistical knowledge and shallow machine learning algorithms have been frequently utilized in MSA [7,11]. As shallow machine learning is limited by feature engineering and massive manual work for data annotation, deep learning becomes the mainstream technique for MSA [12,33,36,43] in an end-to-end manner [38]. In a sense, the key to developing deep learning-based MSA models lies in two aspects: multimodal representation learning and cross-modal fusion.…”

Section: Related Workmentioning

confidence: 99%

Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis

Sun,

Wang,

Jing

et al. 2022

Preprint

View full text Add to dashboard Cite

Existing studies on multimodal sentiment analysis heavily rely on textual modality and unavoidably induce the spurious correlations between textual words and sentiment labels. This greatly hinders the model generalization ability. To address this problem, we define the task of out-of-distribution (OOD) multimodal sentiment analysis. This task aims to estimate and mitigate the bad effect of textual modality for strong OOD generalization. To this end, we embrace causal inference, which inspects the causal relationships via a causal graph. From the graph, we find that the spurious correlations are attributed to the direct effect of textual modality on the model prediction while the indirect one is more reliable by considering multimodal semantics. Inspired by this, we devise a model-agnostic counterfactual framework for multimodal sentiment analysis, which captures the direct effect of textual modality via an extra text model and estimates the indirect one by a multimodal model. During the inference, we first estimate the direct effect by the counterfactual inference, and then subtract it from the total effect of all modalities to obtain the indirect effect for reliable prediction. Extensive experiments show the superior effectiveness and generalization ability of our proposed framework.

show abstract

Exploiting BERT for Multimodal Target Sentiment Classification through Input Space Translation

Cited by 67 publications

References 37 publications

A Survey on Aspect-Based Sentiment Analysis: Tasks, Methods, and Challenges

A Survey on Aspect-Based Sentiment Analysis: Tasks, Methods, and Challenges

Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis

Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis

Contact Info

Product

Resources

About