The complexity dividend: when sophisticated inference matters

Abstract

Animals infer latent properties of the world from noisy and changing observations. Complex, probabilistic approaches to this challenge such as Bayesian inference are accurate but cognitively demanding, relying on extensive working memory and adaptive learning. Simple heuristics are easy to implement but may be less accurate. What is the appropriate balance between complexity and accuracy? We construct a hierarchy of strategies of variable complexity and find a power law of diminishing returns: increasing complexity gives progressively smaller gains in accuracy. The rate of diminishing returns depends systematically on the statistical uncertainty in the world, such that complex strategies do not provide substantial benefits over simple ones when uncertainty is too high or too low. In between, there is a complexity dividend. We translate these theoretical insights into specific predictions about how working memory and adaptivity should be modulated by uncertainty, and we corroborate these predictions in a psychophysical experiment.

Efficient coding of natural scene statistics predicts discrimination thresholds for grayscale textures

“…Even were it possible to accumulate these statistics, adapting to them might not be worth the computational cost of detecting and processing long-range correlations between many intensity values. Understanding the limits of texture adaptation will teach us about the cost-benefit tradeoffs of efficient coding in sensory cortex, in analogy with recently identified cost-benefit tradeoffs in optimal inference (Tavoni et al, 2019). And indeed, although our predictions are in excellent agreement with the data in most cases, we find a few systematic differences that may already be giving us a glimpse of these limits.…”

Efficient coding of natural scene statistics predicts discrimination thresholds for grayscale textures

“…Even were it possible to accumulate these statistics, adapting to them might not be worth the computational cost of detecting and processing long-range correlations between many intensity values. Understanding the limits of texture adaptation will teach us about the cost-benefit tradeoffs of efficient coding in sensory cortex, in analogy with recently identified cost-benefit tradeoffs in optimal inference [36]. And indeed, although our predictions are in excellent agreement with the data in most cases, we find a few systematic differences that may already be giving us a glimpse of these limits.…”

The comparable strategic flexibility of model-free and model-based learning

“…New insights are likely to come from the kinds of information-bottleneck analyses that have been used previously to evaluate complexityoptimality tradeoffs in machine learning (Gilad-Bachrach et al, 2003;Tishby & Zaslavsky, 2015) and biological systems (Palmer et al, 2015). Moreover, this kind of analysis provides a strong framework in which to study notions of bounded rationality (Gigerenzer & Gaissmaier, 2011;Simon, 1955) and resource rational decision-making (Lieder & Griffiths, 2019;Tavoni, Doi, Pizzica, Balasubramanian, & Gold, 2019) that are becoming more prominent in assessing the rationality of human decision-making.…”