Justification of Logarithmic Loss via the Benefit of Side Information

Abstract

Abstract-We consider a natural measure of relevance: the reduction in optimal prediction risk in the presence of side information. For any given loss function, this relevance measure captures the benefit of side information for performing inference on a random variable under this loss function. When such a measure satisfies a natural data processing property, and the random variable of interest has alphabet size greater than two, we show that it is uniquely characterized by the mutual information, and the corresponding loss function coincides with logarithmic loss. In doing so, our work provides a new characterization of mutual information, and justifies its use as a measure of relevance. When the alphabet is binary, we characterize the only admissible forms the measure of relevance can assume while obeying the specified data processing property. Our results naturally extend to measuring causal influence between stochastic processes, where we unify different causal-inference measures in the literature as instantiations of directed information.

Semantic information, autonomous agency and non-equilibrium statistical physics

“…How should we fill in the gap that Shannon explicitly introduced? One kind of approach-common in economics, game theory and statistics-begins by assuming an idealized system that pursues some externally assigned goal, usually formulated as the optimization of an objective function, such as utility [37][38][39][40][41], distortion [36] or prediction error [19,[42][43][44]. Semantic information is then defined as information which helps the system to achieve its goal (e.g.…”