Estimating Graph Dimension with Cross-validated Eigenvalues

Abstract

In applied multivariate statistics, estimating the number of latent dimensions or the number of clusters is a fundamental and recurring problem. One common diagnostic is the scree plot, which shows the largest eigenvalues of the data matrix in decreasing order; the user searches for a "gap" or "elbow" in the decaying eigenvalues; unfortunately, these patterns can hide beneath the bias of the sample eigenvalues. This methodological problem is conceptually difficult because, in many situations, there is only enough signal to detect a subset of the k population dimensions/eigenvectors. In this situation, one could argue that the correct choice of k is the number of detectable dimensions. We alleviate these problems with cross-validated eigenvalues. Under a large class of random graph models, without any parametric assumptions, we provide a p-value for each sample eigenvector. It tests the null hypothesis that this sample eigenvector is orthogonal to (i.e., uncorrelated with) the true latent dimensions. This approach naturally adapts to problems where some dimensions are not statistically detectable. In scenarios where all k dimensions can be estimated, we prove that our procedure consistently estimates k. In simulations and a data example, the proposed estimator compares favorably to alternative approaches in both computational and statistical performance.

OASIS: An interpretable, finite-sample valid alternative to Pearson’sX2for scientific discovery

“…One recent work which shares some similarities with ours is (13), which provides a method for estimating graph dimension with cross-validated eigenvalues. Their method, like ours, is based on splitting the data into two portions, generating embeddings on one part, and testing the signal strength on the held-out portion.…”

Spectral embedding and the latent geometry of multipartite networks

“…Several rank estimation methods, such the cross-validated eigenvector method of Chen et al [6] and as the elbow method of Zhu and Ghodsi [33] are available in the literature. In practice, we suggest obtaining an ambient dimension D using the former, and intrinsic dimensions d1 , .…”