A precise bare simulation approach to the minimization of some distances. Foundations

Abstract

In information theory -as well as in the adjacent fields of statistics, machine learning, artificial intelligence, signal processing and pattern recognition -many flexibilizations of the omnipresent Kullback-Leibler information distance (relative entropy) and of the closely related Shannon entropy have become frequently used tools. To tackle corresponding constrained minimization (respectively maximization) problems by a newly developed dimension-free bare (pure) simulation method, is the main goal of this paper. Almost no assumptions (like convexity) on the set of constraints are needed, within our discrete setup of arbitrary dimension, and our method is precise (i.e., converges in the limit). As a side effect, we also derive an innovative way of constructing new useful distances/divergences. To illustrate the core of our approach, we present numerous examples. The potential for widespread applicability is indicated, too; in particular, we deliver many recent references for uses of the involved distances/divergences and entropies in various different research fields (which may also serve as an interdisciplinary interface).

A Unifying Framework for Some Directed Distances in Statistics

“…This result characterizes D φ pΩ, P q as a rate of escape of P W n from Ω when P does not belong to Ω. We refer to Najim [134], Trashorras & Wintenberger [194], and Broniatowski & Stummer [43] where the latter consider several applications of this result for (deterministic as well as statistical) optimization procedures by bootstrap.…”

“…For tackling the computation of D φ pΩ, P q respectively D φ pΩ, P emp N q on fairly general (e.g. high-dimensional, non-conex and even highly disconnected) constraint sets Ω, a "precise bare simulation" approach has been recently developed by Broniatowski & Stummer [43].…”

“…As far as splitting of the integral e.g. in (43) resp. ( 45) is concerned, notice that the integral `µ1¨λ,λpd ´ν1¨λ,λpd ˘rX s " ş X rf Q pxq ´fP pxqs dλpxq " 1 ´1 " 0 but ş X " fP pxq fQpxq ´1ı dλpxq may be infinite (take e.g.…”

“…Another area where extension of the φ´divergences (i.e. CASM divergences) to signed measures is useful is related to general optimization problems, where one aims at projecting a vector (or a function) on a class of vectors (or a class of functions); we refer to Broniatowski & Stummer [43] for an extensive treatment of such problems in the finite dimensional case.…”