Regularities unseen, randomness observed: Levels of entropy convergence

Abstract

We study how the Shannon entropy of sequences produced by an information source converges to the source's entropy rate. We synthesize several phenomenological approaches to applying information theoretic measures of randomness and memory to stochastic and deterministic processes by using successive derivatives of the Shannon entropy growth curve. This leads, in turn, to natural measures of apparent memory stored in a source and the amounts of information that must be extracted from observations of a source in order for it to be optimally predicted and for an observer to synchronize to it. One consequence of ignoring these structural properties is that the missed regularities are converted to apparent randomness. We demonstrate that this problem arises particularly for small data sets; e.g., in settings where one has access only to short measurement sequences.

Informational and Causal Architecture of Continuous-time Renewal Processes

“…This is equivalent to the more typical random-variable "block" definition of entropy rate [11]: lim δ→∞ H[T 0:δ ]/δ. Similarly, we define the single-measurement entropy rate as:…”

“…To develop it, Sec. II describes the required new notation and definitions that enable extending the -machine framework which is otherwise well understood for discrete-time processes [1,11]. Sections III-V determine the causal and informational architecture of continuous-time renewal processes.…”

Model-free quantification of time-series predictability

“…The information in an observation can be partitioned into two pieces: redundancy and entropy generation [1]. Our approach exploits this decomposition in order to assess how much predictive structure is present in a signal-i.e., where it falls on the complexity spectrum mentioned above.…”

Hidden Markov Models for Automated Protocol Learning

“…In particular, the determinism of the state transitions enables direct calculation of information-theoretic quantities, such as the process's rate of information production (source entropy rate) and the amount of historical information it stores (statistical complexity) [3,14]. Such properties cannot be calculated from an HMM representation that is not an -machine.…”

“…For example, the strings 0110 and 011110 are in its language, but the string 010 is not. It turns out that this process is not equivalent to a Markov Chain of any finite order [14]. In such cases, one must employ a more sophisticated model class such as Hidden Markov Models (HMMs).…”