Consider a Gaussian vector z = (x , y ) , consisting of two subvectors x and y with dimensions p and q respectively. With n independent observations of z, we study the correlation between x and y, from the perspective of the Canonical Correlation Analysis. We investigate the high-dimensional case: both p and q are proportional to the sample size n. Denote by Σuv the population crosscovariance matrix of random vectors u and v, and denote by Suv the sample counterpart. The canonical correlation coefficients between x and y are known as the square roots of the nonzero eigenvalues of the canonical correlation matrix Σ −1 xx ΣxyΣ −1 yy Σyx. In this paper, we focus on the case that Σxy is of finite rank k, i.e. there are k nonzero canonical correlation coefficients, whose squares are denoted by r1 ≥ · · · ≥ r k > 0. We study the sample counterparts of ri, i = 1, . . . , k, i.e. the largest k eigenvalues of the sample canonical correlation matrix S −1 xx SxyS −1 yy Syx, denoted by λ1 ≥ · · · ≥ λ k . We show that there exists a threshold rc ∈ (0, 1), such that for each i ∈ {1, . . . , k}, when ri ≤ rc, λi converges almost surely to the right edge of the limiting spectral distribution of the sample canonical correlation matrix, denoted by d+. When ri > rc, λi possesses an almost sure limit in (d+, 1], from which we can recover ri's in turn, thus provide an estimate of the latter in the high-dimensional scenario. We also obtain the limiting distribution of λi's under appropriate normalization. Specifically, λi possesses Gaussian type fluctuation if ri > rc, and follows Tracy-Widom distribution if ri < rc. Some applications of our results are also discussed.
