Measurement of statistical evidence on an absolute scale following thermodynamic principles

Abstract

Statistical analysis is used throughout biomedical research and elsewhere to assess strength of evidence. We have previously argued that typical outcome statistics (including p values and maximum likelihood ratios) have poor measure-theoretic properties: they can erroneously indicate decreasing evidence as data supporting an hypothesis accumulate; and they are not amenable to calibration, necessary for meaningful comparison of evidence across different study designs, data types, and levels of analysis. We have also previously proposed that thermodynamic theory, which allowed for the first time derivation of an absolute measurement scale for temperature (T), could be used to derive an absolute scale for evidence (E). Here we present a novel thermodynamically based framework in which measurement of E on an absolute scale, for which “one degree” always means the same thing, becomes possible for the first time. The new framework invites us to think about statistical analyses in terms of the flow of (evidential) information, placing this work in the context of a growing literature on connections among physics, information theory, and statistics.

Statistical Evidence Measured on a Properly Calibrated Scale across Nested and Non-nested Hypothesis Comparisons

“…We articulate four BBPs up front (we have described the thought experiments used to motivate the BBPs in detail elsewhere; see, e.g., [5]; here we simply summarize the BBPs themselves). Note that the BBPs described in what follows are specific to this particular set up, which is the one we considered in previous work [4,5]. Extensions to other forms of hypotheses are considered in subsequent sections, albeit still in the context of the binomial model.…”

“…As noted previously, from the start we have viewed S, originally defined as the maximum log LR, as an entropy term; that is, in the original formalism [4] the maximum log LR occupied the place of the term for thermodynamic entropy in the ideal gas EqS. We now explicitly express S as a form of Kullback-Leibler divergence [12] (see Appendix A for details).…”

“…Note too that E is on a proper ratio scale [4,17], so that Figure 5b represents a doubling of the strength of evidence as shown in Figure 5a (and E = 8 represents a doubling again relative to E = 4). This is a unique feature of E compared to all other proposed evidence measures of which we are aware.…”

“…Towards this end, we proposed adapting some of the mathematics of thermodynamics as the basis for an absolute (context-independent) measurement scale for evidence [4]. The result was a new theory of information-dynamics, in which different types of information are conserved and interconverted under principles that resemble the first two laws of thermodynamics, with evidence emerging as a relationship among information types under certain kinds of transformations [5].…”

Information Loss in Binomial Data Due to Data Compression

“…In earlier work we proposed embedding the measurement problem in a novel information dynamic theory [1,2]. Vieland [3] proposed that this theory is grounded in two laws: (1) a form of the likelihood principle, viewed as a conservation principle on a par with the 1st law of thermodynamics; and (2) a law involving information loss incurred by the data analytic process itself through data compression, which she related to (at least one form of) the 2nd law of thermodynamics.…”