Statistical Evidence Measured on a Properly Calibrated Scale across Nested and Non-nested Hypothesis Comparisons

Abstract

Statistical modeling is often used to measure the strength of evidence for or against hypotheses about given data. We have previously proposed an information-dynamic framework in support of a properly calibrated measurement scale for statistical evidence, borrowing some mathematics from thermodynamics, and showing how an evidential analogue of the ideal gas equation of state could be used to measure evidence for a one-sided binomial hypothesis comparison ("coin is fair" vs. "coin is biased towards heads"). Here we take three important steps forward in generalizing the framework beyond this simple example, albeit still in the context of the binomial model. We: (1) extend the scope of application to other forms of hypothesis comparison; (2) show that doing so requires only the original ideal gas equation plus one simple extension, which has the form of the Van der Waals equation; (3) begin to develop the principles required to resolve a key constant, which enables us to calibrate the measurement scale across applications, and which we find to be related to the familiar statistical concept of degrees of freedom. This paper thus moves our information-dynamic theory substantially closer to the goal of producing a practical, properly calibrated measure of statistical evidence for use in general applications.

Statistical Evidence Measured on a Properly Calibrated Scale for Multinomial Hypothesis Comparisons

“…The equation turns out, however, to be quite specific to the one-sided binomial hypothesis contrast (HC) considered here. For example, as we showed in [9], Equation (2) no longer exhibits the BBPs if we apply it to the only slightly more general case of a two-sided binomial HC, "coin is biased toward either heads or tails" vs. "coin is fair". Thus e 1 appears to satisfy Weyl's second stage: It is precise, but overly specific.…”

“…In [9] we gave a formula for evidence that displayed the BBPs for the two-sided binomial HC "coin is biased in either direction" vs. "coin is fair". (In [9] this was classified as a Class II(a) HC: composite vs. simple, nested.)…”

“…In the Discussion, we consider the steps that remain in order to render the theory practical and fully general. Sections 2 and 3 briefly review previously published material [9], although in a somewhat different form in order to make explicit the connections to Weyl's template and to the new results in Sections 4-6.…”

Information Loss in Binomial Data Due to Data Compression

“…One distinctive property of E as defined by this equation is that, as a function of increasing n, it increases in a concave-down manner [2,14]: All other things being equal, E increases more slowly, the stronger it is. This seems to accord with what we mean when we talk about statistical evidence.…”

“…information [2,14]: All other things being equal, E increases more slowly, the stronger it is. This seems to accord with what we mean when we talk about statistical evidence.…”