Using Machine Learning to Identify and Investigate Moderators of Alcohol Use Intervention Effects in Meta-Analyses

Abstract

Aims To illustrate a machine learning-based approach for identifying and investigating moderators of alcohol use intervention effects in aggregate-data meta-analysis. Methods We illustrated the machine learning technique of random forest modeling using data from an ongoing meta-analysis of brief substance use interventions implemented in general healthcare settings. A subset of 40 trials testing brief alcohol interventions (BAIs) was used; these trials provided 344 estimates of post-intervention effects on participants’ alcohol use as well as data on 20 potential moderators of intervention effects. These candidate moderators included characteristics of trial methodology and implementation, intervention design and participant samples. Results The best-fitting random forest model identified 10 important moderators from the pool of 20 candidate moderators. Meta-regression utilizing the selected moderators found that inclusion of prescriptive advice in a BAI session significantly moderated BAI effects on alcohol use. Observed effects were also significantly moderated by several methodological characteristics of trials, including the type of comparison group used, the overall level of attrition and the strategy used to address missing data. In a meta-regression model that included all candidate moderators, fewer coefficients were found to be significant, indicating that the use of a preliminary data reduction technique to identify only important moderators for inclusion in final analyses may have yielded improved statistical power to detect moderation. Conclusions Machine learning methods can be valuable tools for clarifying the influence of trial, intervention and sample characteristics on alcohol use intervention effects, in particular when numerous candidate moderators are available.

Brief alcohol interventions for young adults: Strengthening effects, disentangling mechanisms, and scaling up for impact.

“…Attending to issues of intersectionality will be critical for these efforts, for identifying how individuals’ intersecting identities may differentially expose them to specific risks for high levels of alcohol use (e.g., experiences of discrimination) and opportunities for individual and community resilience, and for identifying different mechanisms of change that may be unique to subpopulations under study. Future efforts should consider exploring the utility of artificial intelligence and machine learning algorithms (Lee et al, 2018; Parr et al, 2022; Schwebel et al, 2022) to assist these efforts to understand when, for whom, and under what conditions these interventions may be optimized for maximal effect across diverse populations. Adaptive trial designs also hold great promise for producing longer term reductions in drinking and alcohol-related consequences, providing an efficient methodology to personalize interventions, while adjusting for the individual’s changes in drinking and other behavioral outcomes over time (Murphy, 2005; Murphy, Collins, et al, 2007; Murphy, Lynch, et al, 2007; Patrick, Lyden, et al, 2021).…”

The associations between discrete emotions and political learning: A cross-disciplinary meta-analysis and systematic review

“…With a high number of coded variables (i.e., possible moderators) accompanied by only a small number of studies and effect sizes included in the models, traditional moderator analysis can be problematic due to multicollinearity and overfitting, as well as increased risk of type 1 errors due to multiple testing (Parr et al, 2022;van Lissa, 2020b). To overcome this problem, we used a machine-learning based exploratory approach to identify relevant moderators, which is specifically suitable for small-scale meta-analyses and can account for dependency in the data (van Lissa, 2020b).…”

“…Both represent an estimation of how much variance will be explained by the model in a new (i.e., unused) dataset, the former one based on data that was not used during bootstrapping while applying the random forest algorithm, and the latter based on the data not used in cross-validation (Parr et al, 2022;van Lissa, 2020b). The final model also estimated variable importance metrics, which can be interpreted analogous to standardized regression coefficients (van Lissa, 2020b) and analysed for significant moderation in multilevel random-effects models.…”