Reliability as Lindley Information

Abstract

This paper introduces a generalized definition of reliability based on Lindley information, which is the mutual information between an observed measure and latent attribute. This definition reduces to the traditional definition of reliability in the case of normal variables, but can be applied to any joint distribution of observed and latent variables. Importantly, unlike traditional definitions of reliability, this formulation of reliability applies to vector-or matrix-valued estimates and summaries of responses, and therefore generalizes reliability to sets of scores and estimates in addition to individual scores and estimates. This formulation also leads to new bounds for reliability, as well as newly reported relationships between reliability and the traditional Fisher information function familiar in item response theory literature. The generalized reliability can be estimated using formulae, or methods used in Bayesian inference such as Markov Chain Monte Carlo (MCMC) depending on the case. Examples based on well-studied datasets are provided, as well as an application to randomly-varying intraindividual covariance structures.Running head: GENERALIZED RELIABILITY 3 The aim of this paper is to introduce a generalized definition of reliability given aswhere ι is the Lindley information (Lindley, 1956), which in this context is the mutual information between an observed variable X and a latent attribute θ:Either X or θ could be multidimensional.Lindley information can be interpreted various ways depending on inferential paradigm and perspective: as an index of the expected change in probability of X and θ under knowledge of their joint distribution, relative to knowledge only of their independent marginal distributions; as the expected change in probability of θ given the data, relative to the prior distribution of θ; as the expected per-observation log-likelihood ratio comparing a model of measure informativeness to a model of measure uninformativeness (Kinney & Atwal, 2014); or as the difference in codelength required to describe the data and θ under a model of informativeness compared to a model of uninformativeness.More generally, Lindley information can be interpreted as the divergence of the joint observed-latent distribution p(X, θ) from the independence distribution p(X)p(θ). As such, it is a special case of f-divergence (Amari, 2009;Gilardoni, 2010;Liese & Vajda, 2006;Sason & Verdu, 2016), which can be defined aswhere f(u) is a convex function f:(0, ∞) → , and p and q are probability densities absolutely continuous with ℝ regard to a reference measure ν. Lindley information is obtained by setting f(u) = u ln u, with p = p(X, θ) and q = p(X)p(θ), and is equivalent to the relative entropy or Kullback-Liebler divergence between p(X, θ) and p(X)p(θ).Like traditional reliability, the generalized definition of reliability implied by Equations 1 and 2 produces a statistic that generally ranges between 0 and 1, and reflects the measurement precision of an indicator, such that values closer to 1 indic...

Reliability, Replicability, and Validity: A Meta-Scientific Analysis

“…These types of considerations might be broadened further to consider the measurand of interest in a particular design. Recently proposed frameworks suggest that reliability can be seen in terms of the information a measurement provides about a parameter of interest (Markon, 2017). Traditionally these parameters of interest have been values of an attribute of interest that vary across individuals (e.g., negative mood, political attitude, memory ability), but it may be advantageous to think in terms of other parameters a measurement might be providing information about, such as values of an attribute that differ across groups.…”

“…This may be especially true if, as has been noted in the cognitive science literature (Hedge, Powell, & Sumner, 2018), many indices of measurement error are focused on individual differences, which might not be directly pertinent to experimental designs focused on maximizing group differences. Quantifying the error of a group difference estimate may be more relevant in such cases than the error in an individual difference estimate (although they can be considered in the same framework; Markon, 2017).…”

The Role of Item Distributions on Reliability Estimation: The Case of Cronbach’s Coefficient Alpha

“…The results from this article should not be interpreted as making the case that coefficient alpha is appropriate only for certain parametric families or distributional forms. Rather, it attempts to elucidate how the information contained within the joint probability distribution is bounded by the lower dimensional marginals and, since reliability attempts to quantify information pertaining to the joint distribution (see Markon, 2017), this same information must obey the restrictions implied by the Fréchet–Hoeffding bounds.…”