VisdaNet: Visual Distillation and Attention Network for Multimodal Sentiment Classification

Abstract

Sentiment classification is a key task in exploring people’s opinions; improved sentiment classification can help individuals make better decisions. Social media users are increasingly using both images and text to express their opinions and share their experiences, instead of only using text in conventional social media. As a result, understanding how to fully utilize them is critical in a variety of activities, including sentiment classification. In this work, we provide a fresh multimodal sentiment classification approach: visual distillation and attention network or VisdaNet. First, this method proposes a knowledge augmentation module, which overcomes the lack of information in short text by integrating the information of image captions and short text; secondly, aimed at the information control problem in the multi-modal fusion process in the product review scene, this paper proposes a knowledge distillation based on the CLIP module to reduce the noise information of the original modalities and improve the quality of the original modal information. Finally, regarding the single-text multi-image fusion problem in the product review scene, this paper proposes visual aspect attention based on the CLIP module, which correctly models the text-image interaction relationship in special scenes and realizes feature-level fusion across modalities. The results of the experiment on the Yelp multimodal dataset reveal that our model outperforms the previous SOTA model. Furthermore, the ablation experiment results demonstrate the efficacy of various tactics in the suggested model.

UsbVisdaNet: User Behavior Visual Distillation and Attention Network for Multimodal Sentiment Classification

“…Later, in Section 6.3 , we demonstrate through experiments that there is an association between user features and sentiment values in the dataset. Therefore, in this section, we make use of the aforementioned user feature theory and incorporate the user features into our previously proposed VisdaNet [ 29 ] model. We improve the “word encoder with word attention” layer in the VisdaNet [ 29 ] model and propose the “word encoder with user behavior attention” in this section.…”

“…Therefore, in this section, we make use of the aforementioned user feature theory and incorporate the user features into our previously proposed VisdaNet [ 29 ] model. We improve the “word encoder with word attention” layer in the VisdaNet [ 29 ] model and propose the “word encoder with user behavior attention” in this section. We call the improved model the User Behavior Visual Distillation and Attention Network (UsbVisdaNet), as shown in Figure 2 .…”

“…Our UsbVisdaNet model is entirely based on improvements to the VisdaNet [ 29 ] model. The architecture of UsbVisdaNet consists of four hierarchical layers, as illustrated in Figure 2 .…”

“…In this section, we follow the approach utilized in VisdaNet [ 29 ] to differentiate between short and long texts based on the number of sentences, denoted as N . Texts with N less than a hyperparameter M are classified as short texts and processed using the knowledge augmentation ( ) algorithm [ 29 ] to augment their information content. Conversely, texts with N greater than M are classified as long texts and processed using the visual distillation ( ) algorithm.…”

“…For short texts, we utilize an advanced image captioning model [ 31 ] to generate image descriptions for the accompanying images in the review, thereby augmenting the text with additional knowledge ( ) [ 29 ]. After applying the algorithm, we can augment the number of sentences in a short review from N to M , resulting in a sentence set : …”