GILE: A Generalized Input-Label Embedding for Text Classification

Abstract

Neural text classification models typically treat output labels as categorical variables which lack description and semantics. This forces their parametrization to be dependent on the label set size, and, hence, they are unable to scale to large label sets and generalize to unseen ones. Existing joint input-label text models overcome these issues by exploiting label descriptions, but they are unable to capture complex label relationships, have rigid parametrization, and their gains on unseen labels happen often at the expense of weak performance on the labels seen during training. In this paper, we propose a new input-label model which generalizes over previous such models, addresses their limitations, and does not compromise performance on seen labels. The model consists of a joint non-linear inputlabel embedding with controllable capacity and a joint-space-dependent classification unit which is trained with cross-entropy loss to optimize classification performance. We evaluate models on full-resource and low-or zero-resource text classification of multilingual news and biomedical text with a large label set. Our model outperforms monolingual and multilingual models which do not leverage label semantics and previous joint input-label space models in both scenarios. 1 Our code is available at

Multilevel Text Alignment with Cross-Document Attention

“…There are three types of similarity learning in NLP. The supervised paradigm differs from typical supervised learning in that training examples are cast into pairwise constraints (Yang and Jin, 2006), as in cross-lingual word embedding learning based on word-level alignments (Faruqui and Dyer, 2014) and zero-shot utterance/document classification (Yazdani and Henderson, 2015;Nam et al, 2016;Pappas and Henderson, 2019) based on utterance/document-level annotations. The unsupervised paradigm aims to learn an underlying low-dimensional space where the relationships between most of the observed data are preserved, as in word embedding learning (Collobert et al, 2011;Mikolov et al, 2013;Pennington et al, 2014;Levy and Goldberg, 2014).…”

Label-Specific Document Representation for Multi-Label Text Classification

“…EX-AM (Du et al, 2018) introduces the interaction mechanism to incorporate word-level matching signals into the text classification task. GILE (Pappas and Henderson, 2019) proposes a joint inputlabel embedding model for neural text classification. Unfortunately, they cannot work well when there is no big difference between label texts.…”

“…However, most of them only focus on document representation but ignore the correlation among labels. Recently, some methods including DXML , EX-AM (Du et al, 2018), SGM (Yang et al, 2018), GILE (Pappas and Henderson, 2019) are proposed to capture the label correlations by exploiting label structure or label content. Although they obtained promising results in some cases, they still cannot work well when there is no big difference between label texts (e.g., the categories Management vs Management moves in Reuters News), which makes them hard to distinguish.…”

Beyond Weight Tying: Learning Joint Input-Output Embeddings for Neural Machine Translation

“…Compared to previous joint input-label models our model is more flexible and not restricted to linear mappings, which have limited expressivity, but uses non-linear mappings modeled similar to energy-based learning networks (Belanger and McCallum, 2016). Perhaps, the most similar embedding model to ours is the one by (Pappas and Henderson, 2018), except for the linear scaling unit which is specific to sigmoidal linear units designed for multi-label classification problems and not for structured prediction, as here.…”