Latent Factor Analysis of Gaussian Distributions under Graphical Constraints

Abstract

The Constrained Minimum Determinant Factor Analysis (CMDFA) setting was motivated by Wyner's common information problem where we seek a latent representation of a given Gaussian vector distribution with the minimum mutual information under certain generative constraints. In this paper, we explore the algebraic structures of the solution space of the CMDFA, when the underlying covariance matrix Σx has an additional latent graphical constraint, namely, a latent star topology. In particular, sufficient and necessary conditions in terms of the relationships between edge weights of the star graph have been found. Under such conditions and constraints, we have shown that the CMDFA problem has either a rank one solution or a rank n − 1 solution where n is the dimension of the observable vector. Further results are given in regards to the solution to the CMDFA with n − 1 latent factors.

Algebraic Properties of Wyner Common Information Solution under Graphical Constraints

“…Next we analyse the solution space of CMDFA and find explicit conditions for both when the solution is rank 1 and when it is not. To start the proceedings we state Theorem 1 given in [7] that gives the necessary and sufficient condition for D * to be the CMDFA solution of the decomposition given in (6).…”

“…As mentioned before, each of X low 1 and X up 1 will produce a corresponding µ from equaion (37) and a set a i , 1 ≤ i ≤ n or equivalently produce a matrix Σ z that decomposes (6). Let X low 1 and X up 1 produce µ low and µ up from equaion (37), the corresponding sets {a low i } n i=1 and {a up i } n i=1 from (35), corresponding matrices Σ low z and Σ up z that decompose (6), and I low , I up be the corresponding mutual information between observed variables and the latent variables respectively.…”

“…Let X low 1 and X up 1 produce µ low and µ up from equaion (37), the corresponding sets {a low i } n i=1 and {a up i } n i=1 from (35), corresponding matrices Σ low z and Σ up z that decompose (6), and I low , I up be the corresponding mutual information between observed variables and the latent variables respectively. Also let Σ star z be the solution to (6) when the CMDFA solution is a star and I star be the corresponding mutual information between the observed variables and the latent factor . Next Theorem analytically shows that each of I low , I up produces better results than I star considered from common information point of view.…”

“…Secondly, Factor Analysis via heuristic local optimization techniques, often based on the expectation maximization algorithm, were computationally tractable but offered no provable performance guarantees. The third and final type are the convex optimization based methods such as Constrained Minimum Trace Factor Analysis (CMTFA) [5] [6] and CMDFA [7]. The motivation behind CMDFA comes from Wyner's common information C(X 1 , X 2 ) which characterizes the minimum amount of common randomness needed to approximate the joint density between a pair of random variables X 1 and X 2 to be C(X 1 , X 2 ) = min P Y X 1 −Y −X 2 I(X 1 , X 2 ; Y ), 1 Md M Hasan, S. Wei and A. Moharrer are with the school of Electrical Engineering and Computer Science, Louisiana State University, Baton Rouge, LA 70803, USA (Email: mhasa15@lsu.edu, swei@lsu.edu, alimoharrer@gmail.com).…”