Upper bound of Bayesian generalization error in non-negative matrix factorization

Abstract

Non-negative matrix factorization ( NMF ) is a new knowledge discovery method that is used for text mining, signal processing, bioinformatics, and consumer analysis. However, its basic property as a learning machine is not yet clarified, as it is not a regular statistical model, resulting that theoretical optimization method of NMF has not yet established. In this paper, we study the real log canonical threshold of NMF and give an upper bound of the generalization error in Bayesian learning. The results show that the generalization error of the matrix factorization can be made smaller than regular statistical models if Bayesian learning is applied.

Variational approximation error in non-negative matrix factorization

“…Therefore, if the prior distribution is gamma distribution, the Main Theorem is valid not only in the case of the probability model and the true distribution are Poisson distributions but also in that of they are normal distributions, exponential distributions, and Bernoulli distributions. Indeed, if the hyperparameters are φ U = φ V = 1, then the prior is strictly positive and bounded, and the upper bound equals the result of [9]. Thus this study gives an extension to the case where the prior distribution is a gamma distribution of the main theorem of the previous work [9].…”

“…In the same way as [9], K(U, V ) ∼ Φ(U, V ) follows. Thus we consider the zero points of Φ(U, V ) and ϕ(U, V ).…”

“…According to our prior researches [9,7], several distributions satisfy that the RLCT of NMF is equal to the absolute of the maximum pole of the following zeta function Specifically, when elements of the data matrix follow normal distribution, Poisson distribution, exponential distribution, and Bernoulli distribution, the behavior of the free energy and the generalization error can be describe using RLCT defined by the zeta function . In these previous studies, the prior distribution was limited to positive and bounded, but when proving that the true distribution and the KL information amount of the probabilistic model have the same RLCT as the square norm error of the matrix, the prior distribution is arbitrary.…”

“…This research was partially supported by the expenses of NTT DATA Mathematical Systems Inc.. According to [9], in the case Lemma 3.1, we consider simultaneous resolution Φ(U, V ) = 0 and ϕ(U, V ) = 0 since {(U, V ) | U V 2 = 0} has intersections with ϕ −1 (0). However, on the other hands, in the case Lemma 3.2, the matrices (U, V ) ∈ Φ −1 (0) cannot have zero elements.…”

Reducing Dimensionality in Text Mining using Conjugate Gradients and Hybrid Cholesky Decomposition

“…Several methods of NMF are discussed here, which include: Semi supervised constrained NMF [19], semisupervised graph based discriminative NMF [20], Bayesian learning approach to reduce the generalization error in upper bound using NMF [21] and update rules [22], sparseness NMF, which provides better characterization of the features [23], sparse unmixing NMF [24], locally weighted sparse graph regularized NMF [25], graph-regularized NMF [26], graph dual regularization [27], multiple graph regularized NMF [28], graph regularized multilayer NMF [29], adaptive graph regularized NMF [30], hyper-graph regularized [31], graph regularization with sparse NMF [32], multi-view NMF [33], extended incremental NMF [34], incremental orthogonal projective NMF [35], correntropy induced metric NMF [36], multi-view NMF [37], patch based NMF [38], MMNMF [39], regularized NMF [40], FR conjugate gradient NMF [41]. However, these methods failed to address the problems associated with non-orthogonality due to the presence of nonnegative elements in NMF.…”