GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Abstract

Every natural text is written in some style. The style is formed by a complex combination of different stylistic factors, including formality markers, emotions, metaphor, etc. Some factors implicitly reflect the author's personality, while others are explicitly controlled by the author's choices in order to achieve some personal or social goal. One cannot form a complete understanding of a text and its author without considering these factors. The factors combine and co-vary in complex ways to form styles. Studying the nature of the covarying combinations sheds light on stylistic language in general, sometimes called crossstyle language understanding. This paper provides a benchmark corpus (xSLUE) with an online platform (http://xslue.com) for crossstyle language understanding and evaluation. The benchmark contains text in 15 different styles and 23 classification tasks. For each task, we provide the fine-tuned classifier for further analysis. Our analysis shows that some styles are highly dependent on each other (e.g., impoliteness and offense), and some domains (e.g., tweets, political debates) are stylistically more diverse than others (e.g., academic manuscripts). We discuss the technical challenges of cross-style understanding and potential directions for future research: crossstyle modeling which shares the internal representation for low-resource or low-performance styles and other applications such as crossstyle generation.

Multi-Task Deep Neural Networks for Natural Language Understanding

“…Adversarial Training Figure 1: The workflow of MT-DNN: train a neural language model on a large amount of unlabeled raw text to obtain general contextual representations; then finetune the learned contextual representation on downstream tasks, e.g. GLUE (Wang et al, 2018); lastly, distill this large model to a lighter one for online deployment. In the later two phrases, we can leverage powerful multi-task learning and adversarial training to further improve performance.…”

“…In this section, we present a comprehensive set of examples to illustrate how to customize MT-DNN for new tasks. We use popular benchmarks from general and biomedical domains, including GLUE (Wang et al, 2018), SNLI (Bowman et al, 2015), SciTail (Khot et al, 2018), SQuAD (Rajpurkar et al, 2016), ANLI (Nie et al, 2019), and biomedical named entity recognition (NER), relation extraction (RE) and question answering (QA) . To make the experiments reproducible, we make all the configuration files publicly available.…”

Language modelling for biological sequences – curated datasets and baselines

“…However, text, like proteins, is often given in a complete form, presenting a possible bidirectionality that cannot be captured by unidirectional language models. As a response, the models BERT (Devlin et al, 2018) and XLNet (Yang et al, 2019) provide bidirectional language modeling objectives by taking inspiration from denoising autoencoders (Vincent et al, 2008), which currently rank state-of-the-art on popular benchmarks such as GLUE (Wang et al, 2018).…”

Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking

“…We evaluated BERT+Entity in the natural 1 TagMe's performance on various benchmark datasets ranges from 37% to 72%. F1 (Kolitsas et al, 2018) language understanding benchmark GLUE (Wang et al, 2018), the question answering (QA) benchmarks SQUAD V2 (Rajpurkar et al, 2018) and SWAG (Zellers et al, 2018), and the machine translation benchmark EN-DE WMT14. We confirm the finding from Zhang et al (2019) that additional entity knowledge is not beneficial for the GLUE benchmark.…”