Evaluating the complexity and falsifiability of psychological models.

Abstract

Understanding model complexity is important for developing useful psychological models. One way to think about model complexity is in terms of the predictions a model makes and the ability of empirical evidence to falsify those predictions. We argue that existing measures of falsifiability have important limitations and develop a new measure. KL-delta uses Kullback–Leibler divergence to compare the prior predictive distributions of models to the data prior that formalizes knowledge about the plausibility of different experimental outcomes. Using introductory conceptual examples and applications with existing models and experiments, we show that KL-delta challenges widely held scientific intuitions about model complexity and falsifiability. In a psychophysics application, we show that hierarchical models with more parameters are often more falsifiable than the original nonhierarchical model. This counters the intuition that adding parameters always makes a model more complex. In a decision-making application, we show that a choice model incorporating response determinism can be harder to falsify than its special case of probability matching. This counters the intuition that if one model is a special case of another, the special case must be less complex. In a memory recall application, we show that using informative data priors based on the serial position curve allows KL-delta to distinguish models that otherwise would be indistinguishable. This shows the value in model evaluation of extending the notion of possible falsifiability, in which all data are considered equally likely, to the more general notion of plausible falsifiability, in which some data are more likely than others.

Good fit is weak evidence of replication: Increasing rigor through prior predictive similarity checking

“…Passing this check would strengthen claims of replication success by ensuring that the data patterns do not resemble the disparate matrices at the top of Figure 1. By directly checking the data patterns, this method is conceptually aligned with specification of a data prior, which has been recently recommended as a tool for stronger theory testing (Vanpaemel, 2020;Villarreal et al, 2023). In the present context, prior specification greatly benefits from reliance on the original findings.…”

The HDI + ROPE decision rule is logically incoherent but we can fix it.

“…1 The three examples above serve to illustrate the ubiquity of nonlinear reparameterizations possible for scientific models across psychology and related fields. Despite a popular preference to represent models using certain "canonical" parameterizations (e.g., Kruschke, 2018), ultimately the epistemic content of a model is not in its parameters but in the distributions it generates over data (Villarreal et al, 2023). There is always an alternative parameterization that could be meaningfully understood; good methods should give us the same conclusions across them all.…”

The HDI+ROPE decision rule is logically incoherent but we can fix it

“…The three examples above serve to illustrate the ubiquity of nonlinear reparameterizations possible for scientific models across psychology and related fields. Despite a popular preference to represent models using certain "canonical" parameterizations (e.g., Kruschke, 2018), ultimately the epistemic content of a model is not in its parameters but in the distributions it generates over data (Villarreal, Etz, & Lee, 2023). There is always an alternative parameterization that could be meaningfully understood; good methods should give us the same conclusions across them all.…”