Retrofitting Structure-aware Transformer Language Model for End Tasks

Fei, Hao; Ren, Yafeng; Ji, Donghong

doi:10.18653/v1/2020.emnlp-main.168

Cited by 36 publications

(27 citation statements)

References 30 publications

(46 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous works for ABSA unfortunately merely make use of the syntactic dependency edge features (i.e., the tree structure) [18,27,31,54]. Without modeling the syntactic dependency labels attached to the dependency arcs, prior studies are limited by treating all word-word relations in the graph equally [16,19,20,23]. Intuitively, the dependency edges with different labels can reveal the relationship more informatively between target aspect and the crucial clues within context, as exemplified in Fig.…”

Section: Syntax Fusion Layermentioning

confidence: 99%

On the Robustness of Aspect-based Sentiment Analysis: Rethinking Model, Data, and Training

Fei

Chua

et al. 2022

ACM Trans. Inf. Syst.

Self Cite

View full text Add to dashboard Cite

Aspect-based sentiment analysis (ABSA) aims at automatically inferring the specific sentiment polarities towards certain aspects of products or services behind the social media texts or reviews, which has been a fundamental application to the real-world society. Within recent decade, ABSA has achieved extraordinarily high accuracy with various deep neural models. However, existing ABSA models with strong in-house performances may fail to generalize to some challenging cases where the contexts are variable, i.e., being low robustness to real-world environment. In this study, we propose to enhance the ABSA robustness by systematically rethinking the bottlenecks from all possible angles, including model, data and training. First, we strengthen the current best-robust syntax-aware models by further incorporating the rich external syntactic dependencies and the labels with aspect simultaneously with a universal-syntax graph convolutional network. In the corpus perspective, we propose to automatically induce high-quality synthetic training data with various types, allowing models to learn sufficient inductive bias for better robustness. Lastly, we based on the rich pseudo data perform adversarial training to enhance the resistance to the context perturbation, and meanwhile employ contrastive learning to reinforce the representations of instances with contrastive sentiments. Extensive robustness evaluations are conducted. The results demonstrate that our enhanced syntax-aware model achieves better robustness performances than all the state-of-the-art baselines. By additionally incorporating our synthetic corpus, the robust testing results are pushed with around 10% accuracy, which are then further improved by installing the advanced training strategies. In-depth analyses are presented for revealing the factors influencing the ABSA robustness.

show abstract

Section: Syntax Fusion Layermentioning

confidence: 99%

On the Robustness of Aspect-based Sentiment Analysis: Rethinking Model, Data, and Training

Fei

Chua

et al. 2022

ACM Trans. Inf. Syst.

Self Cite

View full text Add to dashboard Cite

show abstract

“…1 Our code is available at: https://github.com/JRC1995/Continuous-RvNN Nangia & Bowman, 2018;Choi et al, 2018;Maillard et al, 2019;Havrylov et al, 2019;Shen et al, 2019a) and some of these structure-aware methods (Shen et al, 2019a;Qian et al, 2020) also exhibit better systematicity (Fodor & Pylyshyn, 1988). Notably, even contemporary Transformer-based methods (Vaswani et al, 2017) have benefited from structural biases in multiple natural language tasks (Wang et al, 2019;Fei et al, 2020).…”

Section: Proceedings Of the 38 Th International Conference On Machinementioning

confidence: 99%

“…In recent years, Transformers (Vaswani et al, 2017) have also been extended either to better support tree structured inputs (Shiv & Quirk, 2019;Ahmed et al, 2019) or to have a better inductive bias to induce hierarchical structures by constraining self-attention (Wang et al, 2019;Nguyen et al, 2020;Shen et al, 2021) or by pushing intermediate representations to have constituent information (Fei et al, 2020). However, the fundamental capability of Transformers for composing sequences according to their latent structures in a length-generalizable manner is shown to be lacking (Tran et al, 2018;Shen et al, 2019a;Hahn, 2020).…”

Section: Related Workmentioning

confidence: 99%

Modeling Hierarchical Structures with Continuous Recursive Neural Networks

Chowdhury,

Caragea

2021

Preprint

View full text Add to dashboard Cite

Recursive Neural Networks (RvNNs), which compose sequences according to their underlying hierarchical syntactic structure, have performed well in several natural language processing tasks compared to similar models without structural biases. However, traditional RvNNs are incapable of inducing the latent structure in a plain text sequence on their own. Several extensions have been proposed to overcome this limitation. Nevertheless, these extensions tend to rely on surrogate gradients or reinforcement learning at the cost of higher bias or variance. In this work, we propose Continuous Recursive Neural Network (CRvNN) as a backpropagation-friendly alternative to address the aforementioned limitations. This is done by incorporating a continuous relaxation to the induced structure. We demonstrate that CRvNN achieves strong performance in challenging synthetic tasks such as logical inference (Bowman et al., 2015b) and ListOps (Nangia & Bowman, 2018). We also show that CRvNN performs comparably or better than prior latent structure models on real-world tasks such as sentiment analysis and natural language inference. 1

show abstract

“…The development of ABSA was motivated by this need for granularity, allowing for the extraction of more detailed insights from textual data. ABSA's relevance extends across various domains [10][11][12][13], from enhancing customer service to refining product features based on consumer feedback, thereby playing a pivotal role in data-driven decision-making processes.…”

Section: Introductionmentioning

confidence: 99%

Adversarial Fine-Grained Sentiment Analysis

Rossi,

Al-Masri,

Nasir

et al. 2024

Preprint

View full text Add to dashboard Cite

The digital era has significantly amplified the volume of online reviews, presenting both opportunities and challenges in harnessing insights from consumer feedback. Aspect-Based Sentiment Analysis (ABSA) has emerged as a crucial tool for distilling sentiments from these reviews, providing valuable data for enhancing product and service quality. This study introduces an advanced hybrid model, AdvSentiNet, which integrates adversarial training into the state-of-the-art framework to elevate the precision of sentiment detection at the aspect level. By employing an adversarial network, where a generative model competes against a classifier by crafting highly realistic synthetic samples, we aim to bolster the model's resilience against varied data samples. This innovative approach, unexplored in its entirety within the realm of ABSA, demonstrated remarkable performance improvements on benchmark datasets. For instance, accuracy on the SemEval 2015 dataset escalated from 81.7% to 82.5%, and for the SemEval 2016 dataset, it surged from 84.4% to 87.3%.

show abstract

Retrofitting Structure-aware Transformer Language Model for End Tasks

Cited by 36 publications

References 30 publications

On the Robustness of Aspect-based Sentiment Analysis: Rethinking Model, Data, and Training

On the Robustness of Aspect-based Sentiment Analysis: Rethinking Model, Data, and Training

Modeling Hierarchical Structures with Continuous Recursive Neural Networks

Adversarial Fine-Grained Sentiment Analysis

Contact Info

Product

Resources

About