27 Open Problems in Kolmogorov Complexity

Abstract

Shannon defined the entropy of a random variable M with k values m 1 , . . . , m k having probabilities p 1 , . . . , p k asThis formula can be informally read as follows: the ith message m i brings us log(1/p i ) "bits of information" (whatever this means), and appears with frequency p i , so H is the expected amount of information provided by one random message (one sample of the random variable). Moreover, we can construct an optimal uniquely decodable code that requires about H (at most H + 1, to be exact) bits per message on average, and it encodes ith message by approximately log(1/p i ) bits, following the natural idea to use short codewords for frequent messages. This fits well the informal reading of the formula given above, and it is tempting to say that the ith message "contains log(1/p i ) bits of information." Shannon himself succumbed to this temptation [46, p. 399] when he wrote about entropy estimates and considers Basic English and James Joyces's book "Finnegan's Wake" as two extreme examples of high and low redundancy in English texts. But, strictly speaking, one can speak only of entropies of random variables, not of their individual values, and "Finnegan's Wake" is not a random variable, just a specific string. Can we define the amount of information in individual objects?The algorithmic information theory (AIT) (developed in the 1960s by Kolmogorov and others) tries to go in this direction. A program p (identified with some canonical encoding of it as a string) for a string x is a program that on empty input prints x. AIT defines the amount of information in a finite object (say, a bit string) x as the minimal length of a program for x. This quantity depends on the programming language used, but as Solomonoff, Kolmogorov and Chaitin noted, there exist optimal programming languages (sometimes called universal machines, though this name can be used for many different notions) that make the complexity function minimal up to O(1) additive terms. One then fixes some optimal programming language and uses it to define the Kolmogorov complexity C(x) of a bit string x.Shannon defined also the conditional entropy H(X | Y ) of a random variable X relative to some other random variable Y . Here we assume that X and Y are defined on the same probability space; Shannon's definition can be equivalently rewritten as H(X, Y ) − H(Y ). In algorithmic information theory it is natural