What factors are most important in finding the best model of a psychological process? Comment on Navarro (2018)

Abstract

Psychology research has become increasingly focused on creating formalized models of psychological processes, which can make exact quantitative predictions about observed data that are the result of some unknown psychological process, allowing a better understanding of how psychological processes may actually operate. However, using models to understand psychological processes comes with an additional challenge: how do we select the best model from a range of potential models that all aim to explain the same psychological process? A recent article by Navarro (2018; Computational Brain & Behavior ) provided a detailed discussion on several broad issues within the area of model selection,with Navarro suggesting that "one of the most important functions of a scientific theory is ... to encourage directed exploration of new territory" (p.3), that "understanding how the qualitative patterns in the empirical data emerge naturally from a computational model of a psychological process is often more scientifically useful than presenting a quantified measure of its performance" (p.6), and that "quantitative measures of performance are essentially selecting models based on their ancillary assumptions" (p.6). Here, I provide a critique of several of Navarro's points on these broad issues. In contrast to Navarro, I argue that all possible data should be considered when evaluating a process model (i.e., not just data from novel contexts), that quantitative model selection methods provide a more principled and complete method of selecting between process models than visual assessments of qualitative trends, and that the idea of ancillary assumptions that are not part of the core explanation in the model is a slippery slope to an infinitely flexible model.

A comparison of conflict diffusion models in the flanker task through pseudolikelihood Bayes factors.

“…We only include the full diffusion comparisons within the main text for ease of communication, as a full diffusion version was found to be the superior model for every participant in every data set. We chose the Bayes factor as it closely aligns with our goal of attempting to find the model that provides the best account of the underlying process (Evans, 2019a, 2019c; Gronau & Wagenmakers, 2019). It should also be noted that many different methods—based on different criteria—exist for selecting between competing theoretical models (Evans, 2019a), such as cross-validation methods that select models based on their ability to predict unseen data (Browne, 2000).…”

The Experiment is just as Important as the Likelihood in Understanding the Prior: a Cautionary Note on Robust Cognitive Modeling

“…Although this idea is controversial (see Evans, 2019;Gronau & Wagenmakers, 2019, for counter arguments), we note that the argument made by Navarro (2019) calls for greater investment in generalization and extrapolation. Some ideas behind this are discussed in Kennedy and Gelman (2019), but build upon much older ideas (Shavelson, Webb, & Rowley, 1989).…”

Robust Standards in Cognitive Science

“…Although we agree that both model evaluation and model comparison are important for understanding psychological processes, we disagree that these two categories should form a continuum for selecting between models. Rather, we argue that these different forms of assessment reflect fundamentally different goals of implementing models that answer fundamentally different questions (see Evans, 2019b, for a more in-depth discussion).…”

“…These assessments are usually made through visual assessments of qualitative trends that can be plotted from the data, which are contrasted to the predictions that the model can make for these aspects of the data. It should be noted that model evaluation contains no correction for model flexibility, and therefore, should not be used to answer confirmatory research questions about which models are superior to others, as these comparisons will be biased towards more flexible models (see Roberts & Pashler, 2000;Evans, 2019b, for more detailed discussions). However, model evaluation is ideal for answering research questions about why certain models are found to be superior to others in model comparison, and what further development may be required to create a better explanation of the underlying process, meaning that it is often used in combination with model development.…”