Connecting dots: from local covariance to empirical intrinsic geometry and locally linear embedding

Abstract

Local covariance structure under the manifold setup has been widely applied in the machine learning society. Based on the established theoretical results, we provide an extensive study of two relevant manifold learning algorithms, empirical intrinsic geometry (EIG) and the locally linear embedding (LLE) under the manifold setup. Particularly, we show that without an accurate dimension estimation, the geodesic distance estimation by EIG might be corrupted. Furthermore, we show that by taking the local covariance matrix into account, we can more accurately estimate the local geodesic distance. When understanding LLE based on the local covariance structure, its intimate relationship with the curvature suggests a variation of LLE depending on the "truncation scheme". We provide a theoretical analysis of the variation.

When Locally Linear Embedding Hits Boundary

“…As is discussed in [15], since I p,r (C x + cI p×p ) −1 I p,r can be viewed as the "regularized precision matrix", we can view (ι(z) − ι(x)) I p,r (C x + cI p×p ) −1 I p,r (ι(y) − ι(x)) as the local Mahalanobis distance between z and y, or the distance between the latent variables related to z and y. Thus, when α = 0, the kernel comes from averaging out the pairwise local Mahalanobis distance, and hence depends on the local geometric structure.…”

“…It has been widely applied in different fields, and has been cited more than 12,500 times by the end of 2018 (according to Google Scholar). However, its theoretical justification was only made available at the end of 2017 [29,15]. Essentially, the established theory says that under the manifold setup, LLE has several peculiar behaviors that are very different from those of diffusion-based algorithms, including eigenmap, DM and VDM.…”

“…This asymmetric kernel captures the essence of the currently developed empirical intrinsic geometry framework. Fifth, the kernel depends on the local covariance matrix analysis and the Mahalanobis distance, since it is the mix up of the ordinary kernel and a special kernel depending on the Mahalanobis distance [15].…”

“…While several theoretical properties have been discussed in [29,15], there are more open problems about LLE left. In this paper, we are interested in exploring the asymptotic behavior of LLE when the manifold has a boundary.…”

Explore Intrinsic Geometry of Sleep Dynamics and Predict Sleep Stage by Unsupervised Learning Techniques

“…We mention that this approach leads to a more accurate geodesic distance estimation by a direct hard threshold of the noisy covariance matrix to remove the influence of noise. A more general discussion can be found in [26].…”