NSVQ: Noise Substitution in Vector Quantization for Machine Learning

Abstract

Machine learning algorithms have been shown to be highly effective in solving optimization problems in a wide range of applications. Such algorithms typically use gradient descent with backpropagation and the chain rule. Hence, the backpropagation fails if intermediate gradients are zero for some functions in the computational graph, because it causes the gradients to collapse when multiplying with zero. Vector quantization is one of those challenging functions for machine learning algorithms, since it is a piece-wise constant function and its gradient is zero almost everywhere. A typical solution is to apply the straight through estimator which simply copies the gradients over the vector quantization function in the backpropagation. Other solutions are based on smooth or stochastic approximation. This study proposes a vector quantization technique called NSVQ, which approximates the vector quantization behavior by substituting a multiplicative noise so that it can be used for machine learning problems. Specifically, the vector quantization error is replaced by product of the original error and a normalized noise vector, the samples of which are drawn from a zero-mean, unit-variance normal distribution. We test our proposed NSVQ in three scenarios with various types of applications. Based on the experiments, the proposed NSVQ achieves more accuracy and faster convergence in comparison to the straight through estimator, exponential moving averages, and the MiniBatchKmeans approaches.

A Codebook Compression Method for Vector Quantization Algorithm

“…These methods are covered briefly below. Some studies focused on the learning of the codebook of VQ using machine learning [4] and convolutional neural networks (CNN) [5]. Another approach reported the usage of evolutionary algorithms (EA) [6][7][8] for better codebook design.…”

“…The vector quantization (VQ) algorithm [1][2][3][4][5] is a lossy image compression technique. It divides an image into image blocks and each such image block forms a training vector.…”

A Data-Driven Representation for Sign Language Production

“…This simultaneously helped stabilise training and reduced the required loss functions to two. Vali et al [47] then reduced the required losses to one by introducing the NSVQ technique. Here the vector quantization error is approximated by substituting it for a product of the original error and a normalised noise vector.…”

“…In this paper, we propose creating a data-driven representation of sign language that can be used as a replacement for expensive linguistic annotation. Our approach learns a codebook of motions from continuous 3D pose data using a Noise Substitution Vector Quantization (NSVQ) model [47]. The codebook can be considered the lexicon of our new representation and used to tokenise a continuous pose sequence into a sequence of discrete codes.…”

Construction of order-independent molecular fragments space with vector quantised graph autoencoder

“…Since the number of non-zero components in the codebook is relatively small, the performance of VQGAE latent vectors in virtual screening tasks might be reduced. One way to increase codebook utilisation is to improve the vector quantization operation by applying noise substitution as suggested in 27 or codebook initialisation techniques such as kmeans 28 . In the second example (Figure 9b), for one codebook vector more than 5K different structures of fragments from 42K molecules were extracted.…”