A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representations

Abstract

Learning disentangled representations of textual data is essential for many natural language tasks such as fair classification, style transfer and sentence generation, among others. The existent dominant approaches in the context of text data either rely on training an adversary (discriminator) that aims at making attribute values difficult to be inferred from the latent code or rely on minimising variational bounds of the mutual information between latent code and the value attribute. However, the available methods suffer of the impossibility to provide a fine-grained control of the degree (or force) of disentanglement. In contrast to adversarial methods, which are remarkably simple, although the adversary seems to be performing perfectly well during the training phase, after it is completed a fair amount of information about the undesired attribute still remains. This paper introduces a novel variational upper bound to the mutual information between an attribute and the latent code of an encoder. Our bound aims at controlling the approximation error via the Renyi's divergence, leading to both better disentangled representations and in particular, a precise control of the desirable degree of disentanglement than state-of-the-art methods proposed for textual data. Furthermore, it does not suffer from the degeneracy of other losses in multi-class scenarios. We show the superiority of this method on fair classification and on textual style transfer tasks. Additionally, we provide new insights illustrating various trade-offs in style transfer when attempting to learn disentangled representations and quality of the generated sentence.

InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation

“…Among available contrast measures, the Fisher-Rao distance is parameter-free and thus, it is easy to use in practice while the AB-Divergence achieves better results but requires to select α and β. Future work includes extending our metrics to new tasks such as SLU (Chapuis et al 2020(Chapuis et al , 2021Dinkar et al 2020;Colombo, Clavel, and Piantanida 2021), controlled sentence generation (Colombo et al 2019(Colombo et al , 2021b and multi-modal learning (Colombo et al 2021a;Garcia et al 2019).…”

Learning Disentangled Representations of Negation and Uncertainty

“…. Therefore, this objective minimizes an upper-bound estimate of the MI between each pair of latent spaces, following (Cheng et al, 2020a,b;Colombo et al, 2021).…”

“…Still, no previous work has tested whether negation, uncertainty, and content can be disentangled, as linguistic theory suggests, although previous works have disentangled attributes such as syntax, semantics, and style (Balasubramanian et al, 2021;John et al, 2019;Cheng et al, 2020b;Bao et al, 2019;Hu et al, 2017;Colombo et al, 2021). To fill this gap, we aim to answer the following research questions:…”

Quasi-symbolic explanatory NLI via disentanglement: A geometrical examination

“…Disentangled VAEs in language Early approaches in text disentanglement use VAEs with multiple adversarial losses for style transfer (Hu et al, 2017;John et al, 2019). More recently, Cheng et al (2020) propose a style transfer method which minimizes the mutual information between the latent and the observed variable, while Colombo et al (2021) propose an upper bound of mutual information for fair text classification. Disentanglement of syntactic and semantic information on sentences is explored by , using multiple losses for word ordering and paraphrasing, and by Bao et al (2019) with linearized constituency tree losses.…”