Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks

Aspillaga, Carlos; Carvallo, Andrés; Araujo, Vladimir

doi:10.48550/arxiv.2002.06261

Cited by 4 publications

(4 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Differently to TriRank, our method combines ratings and aspect-based opinions in a KG to learn richer vector representations for both users and items, with which generating personalized recommendations. Furthermore, there have been approaches to automatize text content analysis in the field of medicine [5], along with stress tests [1,2] to evaluate the models performance under different situations.…”

Section: Related Workmentioning

confidence: 99%

Rating and aspect-based opinion graph embeddings for explainable recommendations

Cantador¹,

Carvallo²,

Dı́ez³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

The success of neural network embeddings has entailed a renewed interest in using knowledge graphs for a wide variety of machine learning and information retrieval tasks. In particular, recent recommendation methods based on graph embeddings have shown state-of-the-art performance. In general, these methods encode latent rating patterns and content features. Differently from previous work, in this paper, we propose to exploit embeddings extracted from graphs that combine information from ratings and aspectbased opinions expressed in textual reviews. We then adapt and evaluate state-of-the-art graph embedding techniques over graphs generated from Amazon and Yelp reviews on six domains, outperforming baseline recommenders. Additionally, our method has the advantage of providing explanations that involve the coverage of aspect-based opinions given by users about recommended items.

show abstract

Section: Related Workmentioning

confidence: 99%

Rating and aspect-based opinion graph embeddings for explainable recommendations

Cantador¹,

Carvallo²,

Dı́ez³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Belinkov and Bisk (2017) show that character basedneural machine translation (NMT) models are also prone to synthetic and natural noise even though these model dobetter job to handle out-of-vocabulary issues and learn better morphological representation. Aspillaga et al (2020) evaluated RoBERTa, XL-Net, and BERT in Natural Language Inference (NLI) and Question Answering (QA) tasks. They used BiDAF (Seo et al, 2016) and Match-LSTM (Wang and Jiang, 2016) as baselines to compare stress tests against Transformer-based models.…”

Section: Previous Workmentioning

confidence: 99%

Noisy Text Data: Achilles' Heel of popular transformer based NLP models

Bagla,

Kumar,

Gupta

et al. 2021

Preprint

View full text Add to dashboard Cite

In the last few years, the ML community has created a number of new NLP models based on transformer architecture. These models have shown great performance for various NLP tasks on benchmark datasets, often surpassing SOTA results. Buoyed with this success, one often finds industry practitioners actively experimenting with fine-tuning these models to build NLP applications for industry use cases. However, for most datasets that are used by practitioners to build industrial NLP applications, it is hard to guarantee the presence of any noise in the data. While most transformer based NLP models have performed exceedingly well in transferring the learnings from one dataset to another, it remains unclear how these models perform when fine-tuned on noisy text.We address the open question by Kumar et al. (2020) to explore the sensitivity of popular transformer based NLP models to noise in the text data. We continue working with the noise as defined by them -spelling mistakes & typos (which are the most commonly occurring noise). We show (via experimental results) that these models perform badly on most common NLP tasks namely text classification, textual similarity, NER, question answering, text summarization on benchmark datasets. We further show that as the noise in data increases, the performance degrades. Our findings suggest that one must be vary of the presence of noise in their datasets while fine-tuning popular transformer based NLP models.

show abstract

“…Moreover, using the whole text can often contain redundant and noisy information that can worsen the performance of the model [1], compared to extracting relevant information in the form of aspects, as proposed in this work. Furthermore, there have been approaches to automatize text content analysis in the field of medicine [7], along with stress tests [2,3] to evaluate the models performance under different situations.…”

Section: Review and Text-based Recommendationsmentioning

confidence: 99%

Graphing else matters: exploiting aspect opinions and ratings in explainable graph-based recommendations

Cantador¹,

Carvallo²,

Dı́ez³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The success of neural network embeddings has entailed a renewed interest in using knowledge graphs for a wide variety of machine learning and information retrieval tasks. In particular, current recommendation methods based on graph embeddings have shown state-of-the-art performance. These methods commonly encode latent rating patterns and content features. Different from previous work, in this paper, we propose to exploit embeddings extracted from graphs that combine information from ratings and aspect-based opinions expressed in textual reviews. We then adapt and evaluate state-of-the-art graph embedding techniques over graphs generated from Amazon and Yelp reviews on six domains, outperforming baseline recommenders. Our approach has the advantage of providing explanations which leverage aspect-based opinions given by users about recommended items. Furthermore, we also provide examples of the applicability of recommendations utilizing aspect opinions as explanations in a visualization dashboard, which allows obtaining information about the most and least liked aspects of similar users obtained from the embeddings of an input graph.

show abstract

Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks

Cited by 4 publications

References 28 publications

Rating and aspect-based opinion graph embeddings for explainable recommendations

Rating and aspect-based opinion graph embeddings for explainable recommendations

Noisy Text Data: Achilles' Heel of popular transformer based NLP models

Graphing else matters: exploiting aspect opinions and ratings in explainable graph-based recommendations

Contact Info

Product

Resources

About