Information Theory after 18 Years

Abstract

There are fashions in science. Who in 1900 foresaw the neglect which modern mathematicians accord quaternions and elliptic functions? By the year 2000 information theory may exist only in a few unread definitive treatises, preserved by college librarians as forlorn monuments to misspent lives. To avert this end, information theory must supply something more tangible than vague insights. I hope to show that the gap between theory and practice is closing. CodesThe identifying feature of information theory, the feature which distinguishes it from cybernetics or from other branches of statistical communication theory, is its use of a particular way of "measuring" information. Although earlier authors (1) had suggested various measures of information, C. E. Shannon is the real father of information theory. In a paper which appeared in 1948 (2), he not only defined a reasonable measure for information but also applied it to prove remarkable theorems which provide criteria for evaluating and comparing different communication systems. These theorems came at an advanced stage of the communications art; a radio engineer in 1948 could use amplitude modulation, frequency modulation, phase modulation, single side-band modulation, or pulse code modulation, or he could invent something new without much trouble. The need for good ways of evaluating communication systems was evident, and Shannon's paper received immediate attention.Not all the attention came from communication -engineers, however. Mathematicians found his paper a gold mine of statistical and combinatorial problems, partly because Shannon, writing for engineering readers, had not proved his theorems in the greatest possible generality or with the meticulous rigor that some mathematicians require (3), but mainly because he raised difficult mathematical questions which even today are unanswered. Physicists (4) were interested in a new interpretation of entropy as information. Psychologists found that the new information measure gave a convenient quantitative estimate of the difficulty of certain experimental tasks. Other applications have been to linguistics (5), music (6), cryptography (7), and gambling (8). 320 sites (or, depending on how you count, seven, since South Barrington was listed along with an alternate site in the Chicago area.) These were, Ann Arbor, Michigan; the Brookhaven National Laboratory, Upton, Long Island, N.Y.; Denver, Colorado; Madison, Wisconsin; the Sierra foothills, near Sacramento, California; and South Barrington, or Weston, near Chicago.The proximity of the finalists to major northern universities or research centers inspired senators Sparkman, of Alabama, and Russell, of Georgia, to scriptural sarcasm, with Sparkman declaring, "For unto every one that hath shall be given, and he shall have abundance ... ," and Russell concluding, "But from him that hath not shall be taken away even that which he hath." But though they were joined by Symington, of Missouri, and Mansfield, of Montana, in lamenting the geographic distribution of the fina...

Minimum Energy to Send $k$ Bits Through the Gaussian Channel With and Without Feedback

“…In Fig. 3 we take and compare the bounds 7 The result in [29] differs from (204) by a factor of due to the fact that [29] uses an average over the codebook energy constraint (14) instead of the per-codeword energy constraint in (10). The factor reflects that under the optimal scheme one message has energy zero and all (2 01) others have energy 1 0 .…”

“…Although we are interested in (7), is more suitable for expressing our results and (7) is the solution to (8) Note that (3) also models an infinite-bandwidth continuoustime Gaussian channel without feedback observed over an interval , in which each component corresponds to a different tone in an orthogonal frequency division representation. In that setup, corresponds to the allowed power times , and is the power spectral density of the white Gaussian noise.…”

Processes of skill performance: A foundation for the design and use of training equipment.

“…The earliest, and almost only, comprehensive models were extrapolations from information theory (Gilbert, 1958(Gilbert, , 1966Shannon & Weaver, 1949). This approach received quite a bit of attention in psychology (Attneave, 1959;Fitts, 1954;Hick, 1952;Hyman, 1953;see Johnson, 1967, for bibliography).…”

Information Theory in Living Systems, Methods, Applications, and Challenges

“…Even settling upon a definition for information in biological systems is surprisingly difficult and often contentious. For example, information can be variably expressed in terms of its relationship to a variety of other system traits including (Shannon, 1948;Trincher, 1965;Pierce, 1980;Gilbert, 1966;Szilard, 1929) r randomness, r uncertainty, r predictability, and r thermodynamic entropy.…”