Best Practices in Supervised Machine Learning: A Tutorial for Psychologists

Abstract

Supervised machine learning (ML) is becoming an influential analytical method in psychology and other social sciences. However, theoretical ML concepts and predictive-modeling techniques are not yet widely taught in psychology programs. This tutorial is intended to provide an intuitive but thorough primer and introduction to supervised ML for psychologists in four consecutive modules. After introducing the basic terminology and mindset of supervised ML, in Module 1, we cover how to use resampling methods to evaluate the performance of ML models (bias-variance trade-off, performance measures, k-fold cross-validation). In Module 2, we introduce the nonlinear random forest, a type of ML model that is particularly user-friendly and well suited to predicting psychological outcomes. Module 3 is about performing empirical benchmark experiments (comparing the performance of several ML models on multiple data sets). Finally, in Module 4, we discuss the interpretation of ML models, including permutation variable importance measures, effect plots (partial-dependence plots, individual conditional-expectation profiles), and the concept of model fairness. Throughout the tutorial, intuitive descriptions of theoretical concepts are provided, with as few mathematical formulas as possible, and followed by code examples using the mlr3 and companion packages in R. Key practical-analysis steps are demonstrated on the publicly available PhoneStudy data set ( N = 624), which includes more than 1,800 variables from smartphone sensing to predict Big Five personality trait scores. The article contains a checklist to be used as a reminder of important elements when performing, reporting, or reviewing ML analyses in psychology. Additional examples and more advanced concepts are demonstrated in online materials ( https://osf.io/9273g/ ).

The Causal Cookbook: Recipes for Propensity Scores, G-Computation, and Doubly Robust Standardization

“…Doubly robust estimators provide a partial solution to this problem by combining both nuisance models; they give us two shots to get things right: As long as one of the nuisance models is correctly specified, the resulting estimate does not suffer from misspecification bias. 9 However, this doubly robust property comes at a cost: Although (systematic) bias may be reduced, variance increases in comparison with g-computation (Tan, 2007); thus, we face a bias-variance trade-off (Pargent et al, 2023).…”

Differentiation between atypical anorexia nervosa and anorexia nervosa using machine learning

“…Critically, ML has the potential to elucidate if there are variable interactions that differentiate between categories. Most importantly, ML provides greater flexibility compared to other statistical methodologies previously used in studies evaluating differences between anorexia diagnoses (Pargent et al, 2023).…”

Everything has its price: Foundations of cost-sensitive machine learning and its application in psychology.

“…The basic approach to detect overfitting in ML is to split the sample data into two parts: a training set whose observations are used to train the algorithm and a test set whose observations are predicted to estimate the performance of the trained algorithm on new, unseen data (Yarkoni & Westfall, 2017). Since going into detail about the terminology and foundations of ML would be beyond the scope of this paper, we refer newcomers to Pargent et al (2022) or Yarkoni and Westfall (2017) for introductions more tailored to psychologists; for a special focus on personality research and assessment, see Stachl et al (2020); for a special focus on clinical psychology, see Dwyer et al (2018).…”

“…The outer resampling loop is used to evaluate predictive performance (similar to the case without any hyperparameters) while an inner resampling loop is created in which the tuning procedure as described above is repeated within each of the outer training sets. A more detailed description of nested resampling can be found in Lang et al (2020) or Pargent et al (2022). If accuracy is used in the inner tuning loop, costs are not taken into account when selecting optimal hyperparameter values, even if the Undersampling/oversampling (Elkan, 2001) SMOTE (Chawla et al, 2002;Fernàndez et al, 2018) MetaCost (Domingos, 1999) 3.…”