Vector quantization using information theoretic concepts

Abstract

Abstract. The process of representing a large data set with a smaller number of vectors in the best possible way, also known as vector quantization, has been intensively studied in the recent years. Very efficient algorithms like the Kohonen self-organizing map (SOM) and the Linde Buzo Gray (LBG) algorithm have been devised. In this paper a physical approach to the problem is taken, and it is shown that by considering the processing elements as points moving in a potential field an algorithm equally efficient as the before mentioned can be derived. Unlike SOM and LBG this algorithm has a clear physical interpretation and relies on minimization of a well defined cost function. It is also shown how the potential field approach can be linked to information theory by use of the Parzen density estimator. In the light of information theory it becomes clear that minimizing the free energy of the system is in fact equivalent to minimizing a divergence measure between the distribution of the data and the distribution of the processing elements, hence, the algorithm can be seen as a density matching method.

The Generalized Risk Zone And Observations Selection

“…The Cauchy-Schwartz divergence [9] is a fundamental tool in the GRZ development. It allows the calculation of 'distances' between different probability density functions (pdfs).…”

“…The ITL approach proposed in [4] has overcome this setback by extracting information directly from the observations. Let us consider the Cauchy-Schwartz divergence, between two pdfs p and q, as defined in [9]:…”

“…This methodology was successfully applied in a heart diseases diagnosis problem [17]. Some central tools used in the sequence are in the context of Information Theoretic Learning (ITL) [4], [5], [7]- [9]. ITL is a kernel based methodology that lays hold of information theory concepts.…”

Distributed Vector Quantization Based on Kullback-Leibler Divergence

“…The Linde-Buzo-Gray (LBG) algorithm [3,4] and the self-organization map (SOM) algorithm [5][6][7] are two of the most popular vector quantization algorithms. Based on information theoretic concepts, vector quantization algorithms which aim to minimize the Cauchy-Schwartz (C-S) divergence or the Kullback-Leibler (K-L) divergence between the distributions of the original data and the reproduction vectors have also been devised and have been proven to perform better than the LBG and SOM algorithms [8,9].…”

“…In terms of distributed vector quantization, the LBG and SOM algorithms have been successfully extended to the distributed case in [29] and the proposed distributed LBG and SOM algorithms achieve performances close to that of the corresponding centralized LBG and SOM algorithms, respectively. Since the simulation results in literature on centralized vector quantization have shown that algorithms based on C-S divergence and K-L divergence can achieve better performances than the LBG and SOM algorithms [8,9], it is a natural thought to develop divergence-based vector quantization algorithms in the field of distributed processing. However, the existing divergence-based vector quantization algorithms [8,9] cannot be directly/easily extended to the distributed case due to the lack of data samples in estimating the global data distribution for each individual node (details are provided in the following section).…”