Learning Disentangled Textual Representations via Statistical Measures of Similarity

Abstract

When working with textual data, a natural application of disentangled representations is fair classification where the goal is to make predictions without being biased (or influenced) by sensitive attributes that may be present in the data (e.g., age, gender or race). Dominant approaches to disentangle a sensitive attribute from textual representations rely on learning simultaneously a penalization term that involves either an adversarial loss (e.g., a discriminator) or an information measure (e.g., mutual information). However, these methods require the training of a deep neural network with several parameter updates for each update of the representation model. As a matter of fact, the resulting nested optimization loop is both time consuming, adding complexity to the optimization dynamic, and requires a fine hyperparameter selection (e.g., learning rates, architecture). In this work, we introduce a family of regularizers for learning disentangled representations that do not require training. These regularizers are based on statistical measures of similarity between the conditional probability distributions with respect to the sensitive attributes. Our novel regularizers do not require additional training, are faster and do not involve additional tuning while achieving better results both when combined with pretrained and randomly initialized text encoders.

Unsupervised Layer-wise Score Aggregation for Textual OOD Detection

“…To our knowledge, only a few studies address this issue and attempt to leverage information from all layers. Notably, recent work by Colombo et al (2022a) considers representations obtained by taking the average embedding across the encoder layers. They then apply common OOD detection methods to this new aggregated embedding.…”

“…Choice of the threshold. To select γ, we follow previous work (Colombo et al, 2022a) by selecting an amount of training samples (i.e., "outliers") the detector can wrongfully detect. A classical choice is to set this proportion to 80%.…”

“…Integrated Rank-Weighted depth. Colombo et al (2022a) propose to leverage the Integrated Rank-Weighted (IRW) depth (Ramsay et al, 2019;Staerman et al, 2021a). Similar to the Mahalanobis distance, the IRW data depth measures the centrality/distance of a point to a point cloud.…”

“…Remark 2. It is worth noting that our framework through Agg ∅set or Agg naturally includes previous approaches (Lee et al, 2018;Zhou et al, 2021;Colombo et al, 2022a). For example, the detector of Lee et al (2018) can be obtained by defining Agg ∅ as the minimum of the penultimate line of the matrix S s (x, D N ).…”

“…Softmax-based detectors (Liu et al, 2020;Pearce et al, 2021;Techapanurak et al, 2019) rely on the predicted probabilities to decide whether a sample is OOD. In contrast, hidden-layer-based scores (e.g., cosine similarity, data-depth (Colombo et al, 2022a), or Mahalanobis distance (Lee et al, 2018)) rely on input embedding of the model encoder. In natural language, these methods arbitrarily rely on either the embedding generated by the last layer of encoder (Podolskiy et al, 2021) or on the logits (Wang et al, 2022;Khalid et al, 2022) to compute anomaly scores.…”