Computing f‐divergences and distances of high‐dimensional probability density functions

Abstract

Very often, in the course of uncertainty quantification tasks or data analysis, one has to deal with high‐dimensional random variables. Here the interest is mainly to compute characterizations like the entropy, the Kullback–Leibler divergence, more general f$$ f $$‐divergences, or other such characteristics based on the probability density. The density is often not available directly, and it is a computational challenge to just represent it in a numerically feasible fashion in case the dimension is even moderately large. It is an even stronger numerical challenge to then actually compute said characteristics in the high‐dimensional case. In this regard it is proposed to approximate the discretized density in a compressed form, in particular by a low‐rank tensor. This can alternatively be obtained from the corresponding probability characteristic function, or more general representations of the underlying random variable. The mentioned characterizations need point‐wise functions like the logarithm. This normally rather trivial task becomes computationally difficult when the density is approximated in a compressed resp. low‐rank tensor format, as the point values are not directly accessible. The computations become possible by considering the compressed data as an element of an associative, commutative algebra with an inner product, and using matrix algorithms to accomplish the mentioned tasks. The representation as a low‐rank element of a high order tensor space allows to reduce the computational complexity and storage cost from exponential in the dimension to almost linear.

Multiscale Entanglement Renormalization Ansatz: Causality and Error Correction

“…NISQ superconducting devices suffer with respect to the inclusion of entanglement among qubit triplets and more complex structures than pairs [25], thus leading to a shortrange entanglement which causes a comparable computational capability of tensor network techniques [26]. This successful application for computational complexity reduction in the quantum framework intertwines with widespread applications in data science [27][28][29][30] and hierarchical tensor geometry [31][32][33]. Hierarchical structures typify in machine learning some information filtering procedures, thus naturally framing tensor network renormalization for these applications, as made by means of wavelets [34].…”

Editorial: Tensor numerical methods and their application in scientific computing and data science

“…For the problems of data analysis or uncertainty quantification one should deal with high‐dimensional random variables and the corresponding probability density function. Litvinenko et al 3 propose to present the discretized probability density in the low rank tensor format, which however, makes computationally difficult tasks, when some function of probability density is required which needs the point‐wise representations of a tensor. The arising computational problem becomes tractable when considering the compressed data as a an element of associative commutative algebra with an inner product and using the corresponding matrix algorithms.…”