Representation Learning via Adversarially-Contrastive Optimal Transport

Abstract

In this paper, we study the problem of learning compact (low-dimensional) representations for sequential data that captures its implicit spatiotemporal cues. To maximize extraction of such informative cues from the data, we set the problem within the context of contrastive representation learning and to that end propose a novel objective via optimal transport. Specifically, our formulation seeks a low-dimensional subspace representation of the data that jointly (i) maximizes the distance of the data (embedded in this subspace) from an adversarial data distribution under the optimal transport, a.k.a. the Wasserstein distance, (ii) captures the temporal order, and (iii) minimizes the data distortion. To generate the adversarial distribution, we propose a novel framework connecting Wasserstein GANs with a classifier, allowing a principled mechanism for producing good negative distributions for contrastive learning, which is currently a challenging problem. Our full objective is cast as a subspace learning problem on the Grassmann manifold and solved via Riemannian optimization. To empirically study our formulation, we provide experiments on the task of human action recognition in video sequences. Our results demonstrate competitive performance against challenging baselines.

Contrastive Attraction and Contrastive Repulsion for Representation Learning

“…Tian et al (2019; consider the image views in different modalities and minimize the irrelevant mutual information between them. Most of the works on negative selection observe the merits of using "hard" negative samples, motivating the introduction of additional techniques, such as Mixup and adversarial noise (Bose et al, 2018;Cherian & Aeron, 2020;Li et al, 2020). In a view that not all negative pairs are "true" negatives (Saunshi et al, 2019b), Chuang et al (2020) propose a decomposition of the data distribution to approximate the true negative distribution.…”