Generalizability of statistical prediction from psychological assessment data: An investigation with the MMPI-2-RF.

Abstract

In the present study, the author employed tools and principles from the domain of machine learning to investigate four questions related to the generalizability of statistical prediction in psychological assessment. First, to what extent do predictive methods common to psychology research and machine learning actually tend to predict new data points in new settings? Second, of what practical value is parsimony in applied prediction? Third, what is the most effective way to select model predictors when attempting to maximize generalizability? Fourth, how well do the methods considered compare with one another with respect to prediction generalizability? To address these questions, the author developed various types of predictive models on the basis of Minnesota Multiphasic Personality Inventory (MMPI)-2-RF scales, using multiple prediction criteria, in a calibration inpatient sample, then externally validated those models by applying them to one or two clinical samples from other settings. Model generalizability was then evaluated based on prediction accuracy in the external validation samples. Noteworthy findings from the present study include (a) statistical models generally demonstrated observable performance shrinkage across settings regardless of modeling approach, though they nevertheless tended to retain non-negligible predictive power in new settings; (b) of the modeling approaches considered, regularized (penalized) regression methods appeared to produce the most consistently robust predictions across settings; (c) parsimony appeared more likely to reduce than to enhance model generalizability; and (d) multivariate models whose predictors were selected automatically tended to perform relatively well, often producing substantially more generalizable predictions than models whose predictors were selected based on theory. Public Significance StatementThis study evaluated how well prediction models developed using a variety of approaches accurately generate predictions in new clinical samples, a question rarely considered in psychological assessment research. Among other findings, results suggest that researchers may be able to improve their predictive accuracy across samples by using a more data-driven approach to model construction.

Everything has its Price: Foundations of Cost-Sensitive Learning and its Application in Psychology

“…Traditionally, psychological science has been relying on classical statistical methods such as linear regression models. Unfortunately, their performance in predicting new, unseen data is often poor for various reasons (Menton, 2020;Yarkoni & Westfall, 2017). This lack of generalizability often stems from the fact that models with a large number of predictors fitted to relatively small datasets tend to represent the sample data too closely to enable accurate predictions on new data.…”

“…This lack of generalizability often stems from the fact that models with a large number of predictors fitted to relatively small datasets tend to represent the sample data too closely to enable accurate predictions on new data. This tendency of a model to simply "remember" the sample data is called overfitting (Menton, 2020). The basic approach to detect overfitting in ML is to split the sample data into two parts: a training set whose observations are used to train the algorithm and a test set whose observations are predicted to estimate the performance of the trained algorithm on new, unseen data (Yarkoni & Westfall, 2017).…”

Screening for Depression in Mobile Devices Using Patient Health Questionnaire-9 (PHQ-9) Data: A Diagnostic Meta-Analysis via Machine Learning Methods

“…60,61 It is particularly useful for predicting human behavior, including high-risk behaviors, and it is effective for discriminating psychopathology. 62,63 Machine learning studies are conducted using measures to discriminate psychopathology (eg, prediction suicide ideation, suicide attempt and behaviors, malingering, personality detecting). 61,[63][64][65][66][67][68][69] In addition, studies using PHQ-9 are emerging.…”

“…62,63 Machine learning studies are conducted using measures to discriminate psychopathology (eg, prediction suicide ideation, suicide attempt and behaviors, malingering, personality detecting). 61,[63][64][65][66][67][68][69] In addition, studies using PHQ-9 are emerging. These have strengths in the identification of depression by using machine learning techniques.…”

Detecting suicidal risk using MMPI-2 based on machine learning algorithm

“…For example, some studies use machine learning, the Structured Inventory of Malingered Symptomatology (SIMS) scale 6 , and the Minnesota Multiphasic Personality Inventory-2 (MMPI-2) scale to discriminate malingering to obtain external benefits 7 . Several studies have been conducted on MMPI-2 in particular as it is useful and expandable 8 . Regarding prediction, machine learning technology has advantages in accuracy and scalability compared to conventional statistical approaches 3 .…”