Task-Aware Representation of Sentences for Generic Text Classification

Abstract

State-of-the-art approaches for text classification leverage a transformer architecture with a linear layer on top that outputs a class distribution for a given prediction problem. While effective, this approach suffers from conceptual limitations that affect its utility in few-shot or zero-shot transfer learning scenarios. First, the number of classes to predict needs to be pre-defined. In a transfer learning setting, in which new classes are added to an already trained classifier, all information contained in a linear layer is therefore discarded, and a new layer is trained from scratch. Second, this approach only learns the semantics of classes implicitly from training examples, as opposed to leveraging the explicit semantic information provided by the natural language names of the classes. For instance, a classifier trained to predict the topics of news articles might have classes like "business" or "sports" that themselves carry semantic information. Extending a classifier to predict a new class named "politics" with only a handful of training examples would benefit from both leveraging the semantic information in the name of a new class and using the information contained in the already trained linear layer. This paper presents a novel formulation of text classification that addresses these limitations. It imbues the notion of the task at hand into the transformer model itself by factorizing arbitrary classification problems into a generic binary classification problem. We present experiments in few-shot and zero-shot transfer learning that show that our approach significantly outperforms previous approaches on small training data and can even learn to predict new classes with no training examples at all.

Towards Personalized Intelligence at Scale

“…We aim to design a personalization framework that is generalizable to arbitrary classification tasks without requiring modification to the model architecture. To that end, we draw inspiration from (Halder et al, 2020) and formulate the multiclass classification problem as a series of binary classification tasks:…”

“…We train for 50 epochs (unless noted otherwise) and report F1 scores on the test set. For hyper-parameters, we use a batch size of 16 and a learning rate of 0.02, following the standard in (Halder et al, 2020).…”

“…Zeroshot models have been studied recently as a method for applying pre-trained models to solve new problems without any additional training (Brown et al, 2020a;Raffel et al, 2020;Sanh et al, 2021). While the zeroshot approach is promising, there still exists a large performance gap in the level of accuracy that is needed for production usage (Halder et al, 2020;Wenpeng Yin and Roth, 2019). Thus, the research question remains as to how to enable Personalized Intelligence at a production scale for millions of users and beyond.…”

Don’t Miss the Labels: Label-semantic Augmented Meta-Learner for Few-Shot Text Classification

“…We only require the name of classes rather than manually constructed prompts or templates to convey label semantics. TARS (Halder et al, 2020) also leverages pre-trained language models and label semantics based on binary text classification. However, our method further strengthens generalization ability via meta-learning framework especially in cross-domain and fine-grained cases.…”

MAIR: Framework for mining relationships between research articles, strategies, and regulations in the field of explainable artificial intelligence

“…In the manual labeling process, we achieve 77% agreement between two trained annotators and solve conflicts with a third annotator (the main author of this paper). To classify documents, we use a few-shot learning model based on Task-Aware Representation of Sentences (Halder et al, 2020), implemented in flairNLP (Akbik et al, 2019). We achieve 80.8% accuracy on the holdout set.…”