Robust Modeling in Cognitive Science

Abstract

In an attempt to increase the reliability of empirical findings, psychological scientists have recently proposed a number of changes in the practice of experimental psychology. Most current reform efforts have focused on the analysis of data and the reporting of findings for empirical studies. However, a large contingent of psychologists build models that explain psychological processes and test psychological theories using formal psychological models. Some, but not all, recommendations borne out of the broader reform movement bear upon the practice of behavioral or cognitive modeling. In this article, we consider which aspects of the current reform movement are relevant to psychological modelers, and we propose a number of techniques and practices aimed at making psychological modeling more transparent, trusted, and robust.Cognitive modeling | Reproducibility | Open science | Robustness | Model comparison You never want a serious crisis to go to waste . . . This crisis provides the opportunity for us to do things that you could not before.

A Model Hierarchy for Psychological Science

“…Requiring researchers to motivate the use of their data also minimizes other problematic issues mentioned by Lee et et al (2019) (Allais, 1953;Birnbaum, 2008). Because these failures can be traced back to the model's axioms, it doesn't matter if researchers derived them a priori or stumbled upon when inspecting their results.…”

“…Take a look atFigure 2inLee et al (2019). One can argue that there is no real disagreement between criteria simply because the performance differences are negligible.…”

Towards Replication in Computational Cognitive Modeling: A Machine Learning Perspective

“…Our first persistent issue relates to the effect extensive reproducibility requirements have on different researchers, and applies to most of the preregistration and reporting practices proposed in Lee et al (2019). Diverse levels of interest and expertise might cause great variation in the workload that full reproducibility requires.…”

“…Our final issue again assumes prior challenges have been dealt with; research is fully reproducible and interpetable, adhering to the practices as presented by Lee et al (2019). This leaves dealing with the impact that such a significant 3 For more practical notebooks issues, we recommend watching Grus's 'I don't like notebooks.'…”

The Experiment is just as Important as the Likelihood in Understanding the Prior: a Cautionary Note on Robust Cognitive Modeling

“…Prior predictive checks are a method to robustify Bayesian cognitive modelling that was not covered in detail in the excellent paper of Lee et al (2019). Furthermore, in preparing this paper we were struck that the differences between cognitive modelling and statistical modelling implies that we should be cautious when applying statistical tools for model evaluation to cognitive modelling.…”