A Deep Decomposable Model for Disentangling Syntax and Semantics in Sentence Representation

Abstract

Recently, disentanglement based on a generative adversarial network or a variational autoencoder has significantly advanced the performance of diverse applications in CV and NLP domains. Nevertheless, those models still work on coarse levels in the disentanglement of closely related properties, such as syntax and semantics in human languages. This paper introduces a deep decomposable model based on VAE to disentangle syntax and semantics by using total correlation penalties on KL divergences. Notably, we decompose the KL divergence term of the original VAE so that the generated latent variables can be separated in a more clear-cut and interpretable way. Experiments on benchmark datasets show that our proposed model can significantly improve the disentanglement quality between syntactic and semantic representations for semantic similarity tasks and syntactic similarity tasks.

Exploiting Inductive Bias in Transformers for Unsupervised Disentanglement of Syntax and Semantics with VAEs

“…This separation was performed on various characteristics in text such as style (John et al, 2020;Cheng et al, 2020), sentiment and topic (Xu et al, 2020), or word morphology (Behjati and Henderson, 2021). In works on disentanglement, consequent efforts have been put in the separation between syntax and semantics, whether merely to obtain an interpretable specialization in the embedding space Bao et al, 2019;Ravfogel et al, 2020;, or for controllable generation Chen et al, 2020;Hosking and Lapata, 2021;Li et al, 2021;Hosking et al, 2022). However, all these works rely on syntactic information (constituency parses and PoS tags) or semantic information (paraphrase pairs).…”

Exploiting Inductive Bias in Transformers for Unsupervised Disentanglement of Syntax and Semantics with VAEs

“…This separation was performed on various characteristics in text such as style (John et al, 2020;Cheng et al, 2020), sentiment and topic (Xu et al, 2020), or word morphology (Behjati and Henderson, 2021). In works on disentanglement, consequent efforts have been put in the separation between syntax and semantics, whether merely to obtain an interpretable specialization in the embedding space Bao et al, 2019;Ravfogel et al, 2020;, or for controllable generation Chen et al, 2020;Li et al, 2021). However, all these works rely on syntactic information (constituency parses and PoS tags) or semantic information (paraphrase pairs).…”