Variational Semi-Supervised Aspect-Term Sentiment Analysis via Transformer

Abstract

Aspect-term sentiment analysis (ATSA) is a long-standing challenge in natural language processing. It requires fine-grained semantical reasoning about a target entity appeared in the text. As manual annotation over the aspects is laborious and time-consuming, the amount of labeled data is limited for supervised learning. This paper proposes a semisupervised method for the ATSA problem by using the Variational Autoencoder based on Transformer. The model learns the latent distribution via variational inference. By disentangling the latent representation into the aspect-specific sentiment and the lexical context, our method induces the underlying sentiment prediction for the unlabeled data, which then benefits the ATSA classifier. Our method is classifier-agnostic, i.e., the classifier is an independent module and various supervised models can be integrated. Experimental results are obtained on the SemEval 2014 task 4 and show that our method is effective with different five specific classifiers and outperforms these models by a significant margin.

Learning Discriminative Neural Sentiment Units for Semi-supervised Target-Level Sentiment Classification

“…Many semi-supervised methods have been explored on sentence-level sentiment classification, such as pretraining with Restricted Boltzmann Machine or autoencoder [23,26], auxiliary task learning [24] and adversarial training [25,27]. However, there are only few studies [16,19] on semi-supervised target-level sentiment classification. [19] explored both pretraining and multi-task learning for transferring knowledge from document-level data, which is much less expensive to obtain.…”

“…[19] explored both pretraining and multi-task learning for transferring knowledge from document-level data, which is much less expensive to obtain. [16] used a Transformer-based VAE for pretraining, which modeled the latent distributions via variational inference. However, it failed to distinguish the relevant and irrelevant features with respect to the sentiment.…”

“…This is because they normally require large-scale high-quality labeled data to be well trained, but in practice we have only small target-level labeled data since it is very difficult and costly to collect due to the complex nature of the task, e.g., fine granularity, co-existence of multiple targets in a sentence, and context-sensitive sentiment. Two main methods to address this issue include: (i) generating and incorporating extra sentiment-informative representations by using auxiliary knowledge resources, e.g., sentiment lexicons [17,28]; and (ii) pretraining the embeddings of words or the parameters of networks using large-scale unlabeled data [3,16]. However, both methods can't capture context-dependent sentiment.…”

Interactive Rule Attention Network for Aspect-Level Sentiment Analysis

“…To address this problem, Ma et al [36] propose a novel solution to targeted aspect-based sentiment analysis, which tackles the challenges of both aspect-based sentiment analysis and targeted sentiment analysis by exploiting commonsense knowledge. Cheng et al [37] proposes a semi-supervised method for the ATSA problem by using the Variational Autoencoder based on Transformer. The model learns the latent distribution via variational inference.…”

Incorporating external knowledge into machine learning algorithms for NLP applications

“…where Since the self-attention mechanism in Transformer is able to draw global dependencies between input and output without regard to their distances in the sequence, Transformer is more suitable for capturing long-distance dependencies and allows more parallelization compared with RNN. It has been demonstrated that the Transformer is more effective than RNN in a wide range of NLP tasks [74][75][76][77][78]. Especially, the emerging of pre-trained language models based on the Transformer has lead to a series of breakthroughs and state-of-the-art performances in many NLP applications [3,46,79,80].…”