Best Practices in Supervised Machine Learning: A Tutorial for Psychologists

Abstract

Supervised machine learning (ML) is becoming an influential research method in psychology and other social sciences. However, theoretical ML concepts and predictive modeling techniques are not yet widely taught in psychology programs. This tutorial is intended to provide a low-barrier, non-technical entrance to supervised ML for psychologists in four consecutive modules. After introducing the basic idea of supervised ML, Module I covers performance evaluation of ML models with resampling methods (performance measures, bias-variance tradeoff, k-fold cross-validation). Module II introduces nonlinear, tree-based algorithms, focusing on random forests and their components, regression and classification trees. Module III is about performing empirical benchmark experiments (comparing the performance of several ML algorithms on multiple datasets). Finally, Module IV discusses the interpretation of ML models, including permutation variable importance measures, effect plots (partial dependence plots, individual conditional expectation profiles, accumulated local effect plots), and the concept of model fairness. Throughout the tutorial, intuitive descriptions of theoretical concepts (with as few mathematical formulas as possible) are followed by code examples, using the mlr3 and companion packages in R. Key practical analysis steps are demonstrated on the publicly available PhoneStudy dataset (N = 624), which includes over 1800 variables from smartphone sensing to predict Big Five personality trait scores. The manuscript contains a checklist to be used as a reminder on important aspects when performing, reporting, or reviewing ML analyses in psychology. Additional examples and more advanced concepts are demonstrated in extensive online materials (https://osf.io/9273g/).

Using machine learning to predict student retention from socio-demographic characteristics and app-based engagement metrics

“…Individual decision trees recursively split the feature space (rules to distinguish classes) with the goal to separate the different classes of the criterion (drop out vs. remain in our case). For a detailed description of how individual decision trees operate and translate to a random forest see Pargent, Schoedel & Stachl 80 .…”

Against Algorithmic Exploitation of Human Vulnerabilities

“…The development of such models requires diagnostic, meaning sensitive and privacy protected, information about individuals. Hence there are many challenges and professional requirements that need to be met for safe and ethical handling and development of such models (Pargent, Schoedel, & Stachl, 2022).…”

Everything has its price: Foundations of cost-sensitive machine learning and its application in psychology.

“…The basic approach to detect overfitting in ML is to split the sample data into two parts: a training set whose observations are used to train the algorithm and a test set whose observations are predicted to estimate the performance of the trained algorithm on new, unseen data (Yarkoni & Westfall, 2017). Since going into detail about the terminology and foundations of ML would be beyond the scope of this paper, we refer newcomers to Pargent et al (2022) or Yarkoni and Westfall (2017) for introductions more tailored to psychologists; for a special focus on personality research and assessment, see Stachl et al (2020); for a special focus on clinical psychology, see Dwyer et al (2018).…”