Algorithmic Statistics: Normal Objects and Universal Models

Abstract

Kolmogorov suggested to measure quality of a statistical hypothesis (a model) P for a data x by two parameters: Kolmogorov complexity C(P ) of the hypothesis and the probability P (x) of x with respect to P . The first parameter measures how simple the hypothesis P is and the second one how it fits. The paper [2] discovered a small class of models that are universal in the following sense. Each hypothesis Sij from that class is identified by two integer parameters i, j and for every data x and for each complexity level α there is a hypothesis Sij with j i l(x) of complexity at most α that has almost the best fit among all hypotheses of complexity at most α. The hypothesis Sij is identified by i and the leading i − j bits of the binary representation of the number of strings of complexity at most i. On the other hand, the initial data x might be completely irrelevant to the the number of strings of complexity at most i. Thus Sij seems to have some information irrelevant to the data, which undermines Kolmogorov's approach: the best hypotheses should not have irrelevant information.To restrict the class of hypotheses for a data x to those that have only relevant information, the paper [10] introduced a notion of a strong model for x: those are models for x whose total conditional complexity conditional to x is negligible. An object x is called normal if for each complexity level α at least one its best fitting model of that complexity is strong.In this paper we show that there are "many types" of normal strings (Theorem 10). Our second result states that there is a normal object x such that all its best fitting models Sij are not strong for x. Our last result states that every best fit strong model for a normal object is again a normal object.

Algorithmic Statistics: Forty Years Later

“…However, the arguments from [26] prove the theorem as stated here. This theorem will be proved in Section 7.7.…”

“…In the original theorem from [26] it is claimed only that A is (O((δ + ε+log n) √ n), O((δ +ε+log n) √ n))-normal. However, the arguments from [26] prove the theorem as stated here.…”