Integrating Auxiliary Information in Self-supervised Learning

Abstract

This paper presents to integrate the auxiliary information (e.g., additional attributes for data such as the hashtags for Instagram images) in the self-supervised learning process. We first observe that the auxiliary information may bring us useful information about data structures: for instance, the Instagram images with the same hashtags can be semantically similar. Hence, to leverage the structural information from the auxiliary information, we present to construct data clusters according to the auxiliary information. Then, we introduce the Clustering InfoNCE (Cl-InfoNCE) objective that learns similar representations for augmented variants of data from the same cluster and dissimilar representations for data from different clusters. Our approach contributes as follows: 1) Comparing to conventional self-supervised representations, the auxiliary-information-infused self-supervised representations bring the performance closer to the supervised representations; 2) The presented Cl-InfoNCE can also work with unsupervised constructed clusters (e.g., k-means clusters) and outperform strong clustering-based self-supervised learning approaches, such as the Prototypical Contrastive Learning (PCL) method; 3) We show that Cl-InfoNCE may be a better approach to leverage the data clustering information, by comparing it to the baseline approach -learning to predict the clustering assignments with cross-entropy loss. For analysis, we connect the goodness of the learned representations with the statistical relationships: i) the mutual information between the labels and the clusters and ii) the conditional entropy of the clusters given the labels. * equal contribution Preprint. Under review.

Conditional Contrastive Learning with Kernel

“…Recent literature (Robinson et al, 2020;Tsai et al, 2021a;c) has modified the InfoNCE objective to achieve different learning goals by sampling positive or negative pairs under conditioning variable Z (and its outcome z). These different conditional contrastive learning frameworks have one common technical challenge: the conditional sampling procedure.…”

“…Weakly Supervised Contrastive Learning. Tsai et al (2021a) consider the auxiliary information from data (e.g., annotation attributes of images) as a weak supervision signal and propose a contrastive objective to incorporate the weak supervision in the representations. This work is motivated by the argument that the auxiliary information implies semantic similarities.…”

“…We refer to these frameworks as unconditional contrastive learning, in contrast to our paper's focus -conditional contrastive learning, which considers contrastive objectives that take additional conditioning variables into account. Such conditioning variables can be sensitive information from data (Tsai et al, 2021c), auxiliary information from data (Tsai et al, 2021a), downstream labels (Khosla et al, 2020), or data's embedded features (Robinson et al, 2020;Kalantidis et al, 2020). It is worth noting that, with additional conditioning variables, the conditional contrastive frameworks extend the self-supervised learning settings to the weakly supervised learning (Tsai et al, 2021a) or the supervised learning setting (Khosla et al, 2020).…”

“…In conditional contrastive learning, positive and negative pairs are constructed conditioned on specific variables. The conditioning variables can be downstream labels (Khosla et al, 2020), sensitive attributes (Tsai et al, 2021c), auxiliary information (Tsai et al, 2021a), or data embedding features (Robinson et al, 2020;. For example, in fair contrastive learning (Tsai et al, 2021c), conditioning on variables such as gender or race, is performed to remove undesirable information from the learned representations.…”

“…Conditioning is achieved by constructing negative pairs from the same gender. As a second example, in weakly-supervised contrastive learning (Tsai et al, 2021a), the aim is to include extra information in the learned representations. This extra information could be, for example, some freely available attributes for images collected from social media.…”