Sufficient Dimension Reduction via Squared-Loss Mutual Information Estimation

Abstract

The goal of sufficient dimension reduction in supervised learning is to find the lowdimensional subspace of input features that is 'sufficient' for predicting output values. In this paper, we propose a novel sufficient dimension reduction method using a squaredloss variant of mutual information as a dependency measure. We utilize an analytic approximator of squared-loss mutual information based on density ratio estimation, which is shown to possess suitable convergence properties. We then develop a natural gradient algorithm for sufficient subspace search. Numerical experiments show that the proposed method compares favorably with existing dimension reduction approaches.

Unsupervised Dimension Reduction via Least-Squares Quadratic Mutual Information

“…A local maximizer may be obtained by a gradient-projection method or a natural gradient method [6], but we consider a computationally more efficient approach based on [8], which is described below.…”

Registration of infrared transmission images using squared-loss mutual information

“…Thus, SMI can be used as an alternative to MI for evaluating statistical dependency, without suffering the "log" problem. Furthermore, thanks to the simple squared-difference expression of SMI, it can be analytically approximated from samples in a statistically optimal way, and this analytic SMI approximator allows explicit computation of its derivative [10]. This is a highly useful property in image registration, because eventually we want to register images so that SMI is maximized with respect to some image transformation parameters.…”

On Kernel Parameter Selection in Hilbert-Schmidt Independence Criterion

“…Measuring statistical independence between random variables is an important challenge in machine learning, because it can be used for various purposes such as feature selection [17], [24], feature extraction [22], [25], clustering [9], [16], [21], statistical independence test [7], [19], independent component analysis [15], [23], object matching [13], [27], and causal inference [11], [26].…”

“…The basic idea of LSMI is to approximate the ratio of a joint density over the product of marginal densities directly in a single-shot process, allowing us to avoid density estimation systematically [20]. LSMI was shown to possess a superior non-parametric convergence property [22] and optimal numerical stability [8]. Furthermore, LSMI is equipped with cross-validation that can be used for objectively determining kernel parameters.…”