Fractal approach for determining the optimal number of topics in the field of topic modeling.

The 5th International Electronic Conference on Entropy and Its Applications

Pashakhin

2019

In practice, the critical step in building machine learning models of big data (BD) is costly in terms of time and the computing resources procedure of parameter tuning with a grid search. Due to the size, BD are comparable to mesoscopic physical systems. Hence, methods of statistical physics could be applied to BD. The paper shows that topic modeling demonstrates self-similar behavior under the condition of a varying number of clusters. Such behavior allows using a renormalization technique. The combination of a renormalization procedure with the Rényi entropy approach allows for fast searching of the optimal number of clusters. In this paper, the renormalization procedure is developed for the Latent Dirichlet Allocation (LDA) model with a variational Expectation-Maximization algorithm. The experiments were conducted on two document collections with a known number of clusters in two languages. The paper presents results for three versions of the renormalization procedure: (1) a renormalization with the random merging of clusters, (2) a renormalization based on minimal values of Kullback-Leibler divergence and (3) a renormalization with merging clusters with minimal values of Rényi entropy. The paper shows that the renormalization procedure allows finding the optimal number of topics 26 times faster than grid search without significant loss of quality.

show abstract

Section: Entropic Approach For Determining the Optimal Number Of Topicsmentioning

confidence: 93%

Fast Tuning of Topic Models: An Application of Rényi Entropy and Renormalization Theory

The 5th International Electronic Conference on Entropy and Its Applications

Pashakhin

2019

show abstract

“…Moreover, Renyi entropy is closely related to renormalization procedures [ 43 ]. However, the works on the investigation of fractal behavior in the models of soft clustering (TM, in particular) and, consequently, on renormalization procedures of such models are very limited [ 16 , 17 , 18 ].…”

Section: Methodsmentioning

confidence: 99%

“…In [ 16 ], the multifractal approach is applied to the analysis of the behavior of topic models. This work proposes to consider the results of TM as an embedding of the space of words into a lattice of size

, where T is the number of topics (the number of columns in matrix

), and W is the number of unique words (the number of rows in matrix

).…”

Section: Methodsmentioning

confidence: 99%

“…If the size of the vocabulary is fixed, then the size of cells is determined by the number of topics, moreover, if

, then the size of cells tends to zero. Reference [ 16 ] investigates the behavior of the density-of-states function under the variation of the cell size through the ‘box counting’ algorithm. Fractal approach to the analysis of the results of TM allows for detecting areas of self-similarity of the word distribution density function when the number of topics is varied.…”

Section: Methodsmentioning

confidence: 99%

“…In this work, we propose a way to overcome this limitation, at least for some models. While testing the Renyi entropy approach, we found out that some functions of the output of TM (namely, density-of-states function [ 16 ] and partition function, which is defined in Section 2.3 ), which are used for Renyi entropy calculation, possess self-similar behavior. This finding led us to think about the possibility of using the renormalization technique when calculating Renyi entropy.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Renormalization Analysis of Topic Models

2020

Entropy

In practice, to build a machine learning model of big data, one needs to tune model parameters. The process of parameter tuning involves extremely time-consuming and computationally expensive grid search. However, the theory of statistical physics provides techniques allowing us to optimize this process. The paper shows that a function of the output of topic modeling demonstrates self-similar behavior under variation of the number of clusters. Such behavior allows using a renormalization technique. A combination of renormalization procedure with the Renyi entropy approach allows for quick searching of the optimal number of topics. In this paper, the renormalization procedure is developed for the probabilistic Latent Semantic Analysis (pLSA), and the Latent Dirichlet Allocation model with variational Expectation–Maximization algorithm (VLDA) and the Latent Dirichlet Allocation model with granulated Gibbs sampling procedure (GLDA). The experiments were conducted on two test datasets with a known number of topics in two different languages and on one unlabeled test dataset with an unknown number of topics. The paper shows that the renormalization procedure allows for finding an approximation of the optimal number of topics at least 30 times faster than the grid search without significant loss of quality.

show abstract

Renormalization Approach to the Task of Determining the Number of Topics in Topic Modeling

Advances in Intelligent Systems and Computing

2020