Nonlinear Canonical Correlation Analysis:A Compressed Representation Approach

Abstract

Canonical Correlation Analysis (CCA) is a linear representation learning method that seeks maximally correlated variables in multi-view data. Nonlinear CCA extends this notion to a broader family of transformations, which are more powerful in many real-world applications. Given the joint probability, the Alternating Conditional Expectation (ACE) algorithm provides an optimal solution to the nonlinear CCA problem. However, it suffers from limited performance and an increasing computational burden when only a finite number of samples is available. In this work, we introduce an information-theoretic compressed representation framework for the nonlinear CCA problem (CRCCA), which extends the classical ACE approach. Our suggested framework seeks compact representations of the data that allow a maximal level of correlation. This way, we control the trade-off between the flexibility and the complexity of the model. CRCCA provides theoretical bounds and optimality conditions, as we establish fundamental connections to rate-distortion theory, the information bottleneck and remote source coding. In addition, it allows a soft dimensionality reduction, as the compression level is determined by the mutual information between the original noisy data and the extracted signals. Finally, we introduce a simple implementation of the CRCCA framework, based on lattice quantization.

An Approach to Canonical Correlation Analysis Based on Rényi’s Pseudodistances

“…There is an extensive bibliography addressing non-linear CCA. Without wishing to cite all the existing literature on the topic, we would like to mention some interesting works on the subject: [7] (Chapter 6), [8][9][10][11] and references therein.…”

The Compound Information Bottleneck Outlook

“…• Minimal Correlation class: P X Y = {P XY : E [XY] ≥ ρ 1 }. This class is motivated by the compressed representation canonical correlation analysis (CRCCA) [11]. The interpretation is similar to the privacy funnel case, only here the correlation replaces the mutual information as a measure of statistical dependence.…”