Renormalization Analysis of Topic Models

Abstract

In practice, to build a machine learning model of big data, one needs to tune model parameters. The process of parameter tuning involves extremely time-consuming and computationally expensive grid search. However, the theory of statistical physics provides techniques allowing us to optimize this process. The paper shows that a function of the output of topic modeling demonstrates self-similar behavior under variation of the number of clusters. Such behavior allows using a renormalization technique. A combination of renormalization procedure with the Renyi entropy approach allows for quick searching of the optimal number of topics. In this paper, the renormalization procedure is developed for the probabilistic Latent Semantic Analysis (pLSA), and the Latent Dirichlet Allocation model with variational Expectation–Maximization algorithm (VLDA) and the Latent Dirichlet Allocation model with granulated Gibbs sampling procedure (GLDA). The experiments were conducted on two test datasets with a known number of topics in two different languages and on one unlabeled test dataset with an unknown number of topics. The paper shows that the renormalization procedure allows for finding an approximation of the optimal number of topics at least 30 times faster than the grid search without significant loss of quality.

A Latent Topic Analysis and Visualization Framework for Category-Level Target Promotion in the Supermarket

“…Thus, the complex system's entropy differences can be measured to discover when an information maximum is reached. Koltcov's research considers entropy as negative information; thus, the maximum entropy corresponds to the minimum of information [14,15]. Therefore, the value of T corresponding to the smallest entropy value can be considered the "true number of topics," representing the maximum valid information generated by the topic model.…”

“…Based on Koltcovs' research [14], the Renyi entropy can be expressed as follows: where q = 1∕T is called the deformation parameter and T is the number of topics. Z q is the partition function of a topic solution which is shown as below:…”

“…In addition to the application of Renyi entropy, this study also employs a renormalization technique to accelerate the computational speed for a fast approximation of Renyi entropy and to search the optimal number of topics [14]. The procedure of implementing renormalization for the LDA model can be explained as follows:…”

“…Especially when using supermarket data, the number of topics representing customer's purchase tendency will directly impact understanding customer behavior and the decisions about which products should be promoted. While most of the topic models require the user to select this number manually, Koltcov's research about the application of Renyi entropy on topic models provides an automated, time-saving solution [14,15]. Therefore, this study also implements this method to automatically search for the optimal number of topics in the POS data.…”

Selection of the Optimal Number of Topics for LDA Topic Model—Taking Patent Policy Analysis as an Example

“…From a text mining perspective, topics in a text corpus can be viewed as probability distributions of terms present in the corpus or clusters that define weights for those terms [1,2]. The LDA model (Latent Dirichlet Allocation) uses a probability distribution model to generate topics [3,4]. The principle is that a document is assumed to be generated from multiple topics according to a certain random probability distribution, and each topic is composed of words according to a random probability distribution.…”