Learning Independent Features with Adversarial Nets for Non-linear ICA

Brakel, Philémon; Bengio, Yoshua

doi:10.48550/arxiv.1710.05050

Cited by 26 publications

(45 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, both of these bounds require a large number of negative samples and as a result, recent methods either train with extremely large batch-sizes [7,40] or an additional memory-bank of negative samples [8,48]. Unlike these works, we estimate mutual information using a Jensen-Shannon Divergence (JSD) bound, similar to formulations used for generative modeling [36] and source separation [5]. This bound on mutual information is derived by replacing the KL-divergence in equation 2 with the Jensen-Shannon divergence.…”

Section: Mutual Information Maximizationmentioning

confidence: 99%

CLIP-Lite: Information Efficient Visual Representation Learning from Textual Annotations

Shrivastava¹,

Selvaraju²,

Naik³

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose CLIP-Lite, an information efficient method for visual representation learning by feature alignment with textual annotations. Compared to the previously proposed CLIP model, CLIP-Lite requires only one negative imagetext sample pair for every positive image-text sample during the optimization of its contrastive learning objective. We accomplish this by taking advantage of an information efficient lower-bound to maximize the mutual information between the two input modalities. This allows CLIP-Lite to be trained with significantly reduced amounts of data and batch sizes while obtaining better performance than CLIP. We evaluate CLIP-Lite by pretraining on the COCO-Captions dataset and testing transfer learning to other datasets. CLIP-Lite obtains a +15.4% mAP absolute gain in performance on Pascal VOC classification, and a +22.1% top-1 accuracy gain on ImageNet, while being comparable or superior to other, more complex, textsupervised models. CLIP-Lite is also superior to CLIP on image and text retrieval, zero-shot classification, and visual grounding. Finally, by performing explicit image-text alignment during representation learning, we show that CLIP-Lite can leverage language semantics to encourage biasfree visual representations that can be used in downstream tasks.

show abstract

Section: Mutual Information Maximizationmentioning

confidence: 99%

CLIP-Lite: Information Efficient Visual Representation Learning from Textual Annotations

Shrivastava¹,

Selvaraju²,

Naik³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Several methods for learning disentangled representations have been recently proposed, however lacking identifiability guarantees, most of which are based on a variational autoencoder (VAE, Kingma and Welling [33]) formulation, and decompositions of the VAE objective, for example [19,31,8,7,35,6,14]. GAN-based approaches for disentanglement have been proposed as well [8,5].…”

Section: Related Workmentioning

confidence: 99%

“…We then generate the observed data via linear mixing. We consider one of the latent components as the condition and train the autoencoder to reconstruct the observed data, while obtaining code which is independent of the condition using the regression objective (5).…”

Section: D Analysis Demonstrationmentioning

confidence: 99%

Discovery of Single Independent Latent Variable

Shaham¹,

Svirsky²,

Katz³

et al. 2021

Preprint

View full text Add to dashboard Cite

Latent variable discovery is a central problem in data analysis with a broad range of applications in applied science. In this work, we consider data given as an invertible mixture of two statistically independent components, and assume that one of the components is observed while the other is hidden. Our goal is to recover the hidden component. For this purpose, we propose an autoencoder equipped with a discriminator. Unlike the standard nonlinear ICA problem, which was shown to be non-identifiable, in the special case of ICA we consider here, we show that our approach can recover the component of interest up to entropy-preserving transformation. We demonstrate the performance of the proposed approach on several datasets, including image synthesis, voice cloning, and fetal ECG extraction.

show abstract

“…Different from above methods, in our approach we use Jensen-Shannon divergence based mutual information estimation, similar to the formulations in (Nowozin, Cseke, and Tomioka 2016) and (Brakel and Bengio 2017),…”

Section: Mutual Information Maximizationmentioning

confidence: 99%

Estimating and Maximizing Mutual Information for Knowledge Distillation

Shrivastava¹,

Qi²,

Ordóñez³

2021

Preprint

View full text Add to dashboard Cite

Knowledge distillation is a widely used general technique to transfer knowledge from a teacher network to a student network. In this work, we propose Mutual Information Maximization Knowledge Distillation (MIMKD). Our method uses a contrastive objective to simultaneously estimate and maximize a lower bound on the mutual information between intermediate and global feature representations from the teacher and the student networks. Our method is flexible, as the proposed mutual information maximization does not impose significant constraints on the structure of the intermediate features of the networks. As such, we can distill knowledge from arbitrary teachers to arbitrary students. Our empirical results show that our method outperforms competing approaches across a wide range of student-teacher pairs with different capacities, with different architectures, and when student networks are with extremely low capacity. We are able to obtain 74.55% accuracy on CIFAR100 with a Shuf-flenetV2 from a baseline accuracy of 69.8% by distilling knowledge from ResNet50.

show abstract

Learning Independent Features with Adversarial Nets for Non-linear ICA

Cited by 26 publications

References 12 publications

CLIP-Lite: Information Efficient Visual Representation Learning from Textual Annotations

CLIP-Lite: Information Efficient Visual Representation Learning from Textual Annotations

Discovery of Single Independent Latent Variable

Estimating and Maximizing Mutual Information for Knowledge Distillation

Contact Info

Product

Resources

About