It was mentioned by Kolmogorov (1968, IEEE Trans. Inform. Theory 14, 662 664) that the properties of algorithmic complexity and Shannon entropy are similar. We investigate one aspect of this similarity. Namely, we are interested in linear inequalities that are valid for Shannon entropy and for Kolmogorov complexity. It turns out that (1) all linear inequalities that are valid for Kolmogorov complexity are also valid for Shannon entropy and vice versa; (2) all linear inequalities that are valid for Shannon entropy are valid for ranks of finite subsets of linear spaces; (3) the opposite statement is not true; Ingleton's inequality (1971,``Combinatorial Mathematics and Its Applications,'' pp. 149 167. Academic Press, San Diego) is valid for ranks but not for Shannon entropy; (4) for some special cases all three classes of inequalities coincide and have simple description. We present an inequality for Kolmogorov complexity that implies Ingleton's inequality for ranks; another application of this inequality is a new simple proof of one of Ga cs Ko rner's results on common information (1973, Problems Control Inform. Theory 2, 149 162).

In 1974 Kolmogorov proposed a nonprobabilistic approach to statistics and model selection. Let data be finite binary strings and models be finite sets of binary strings. Consider model classes consisting of models of given maximal (Kolmogorov) complexity. The "structure function" of the given data expresses the relation between the complexity level constraint on a model class and the least log-cardinality of a model in the class containing the data. We show that the structure function determines all stochastic properties of the data: for every constrained model class it determines the individual best-fitting model in the class irrespective of whether the "true" model is in the model class considered or not. In this setting, this happens with certainty, rather than with high probability as is in the classical case. We precisely quantify the goodness-of-fit of an individual model with respect to individual data. We show that-within the obvious constraints-every graph is realized by the structure function of some data. We determine the (un)computability properties of the various functions contemplated and of the "algorithmic minimal sufficient statistic."Index Termsconstrained minimum description length (ML) constrained maximum likelihood (MDL) constrained best-fit model selection computability lossy compression minimal sufficient statistic non-probabilistic statistics Kolmogorov complexity, Kolmogorov Structure function prediction sufficient statistic

Abstract. In this paper we prove a countable set of non-Shannon-type linear information inequalities for entropies of discrete random variables, i.e., information inequalities which cannot be reduced to the "basic" inequality I(X : Y |Z) ≥ 0. Our results generalize the inequalities of Z. Zhang and R. Yeung (1998) who found the first examples of non-Shannon-type information inequalities.

No abstract

A nonnegative martingale with initial value equal to one measures evidence against a probabilistic hypothesis. The inverse of its value at some stopping time can be interpreted as a Bayes factor. If we exaggerate the evidence by considering the largest value attained so far by such a martingale, the exaggeration will be limited, and there are systematic ways to eliminate it. The inverse of the exaggerated value at some stopping time can be interpreted as a p-value. We give a simple characterization of all increasing functions that eliminate the exaggeration.

Abstract

Given a machine U, a c-short program for x is a string p such that U(p) = x and the length of p is bounded by c + (the length of a shortest program for x). We show that for any standard Turing machine, it is possible to compute in polynomial time on input x a list of polynomial size guaranteed to contain a O log |x| -short program for x. We also show that there exists a computable function that maps every x to a list of size |x| 2 containing a O 1 -short program for x. This is essentially optimal because we prove that for each such function there is a c and infinitely many x for which the list has size at least c|x| 2 . Finally we show that for some standard machines, computable functions generating lists with 0-short programs, must have infinitely often list sizes proportional to 2 |x| .

Starting with the paper of Baker, Gill and Solovay BGS 75] in complexity theory, many results have been proved which separate certain relativized complexity classes or show that they have no complete language. All results of this kind were, in fact, based on lower bounds for boolean decision trees of a certain type or for machines with polylogarithmic restrictions on time. The following question arises: Are these methods of proving \relativized" results universal? In the rst part of the present paper we propose a general framework in which assertions of universality of this kind may be formulated and proved as convenient criteria. Using these criteria we obtain, as easy consequences of the known results on boolean decision trees, some new \relativized" results and new proofs of some known results. In the second part of the present paper we apply these general criteria to many particular cases. For example, for many of the complexity classes studied in the literature all relativizable inclusions between the classes are found.