Investigation of a Data Split Strategy Involving the Time Axis in Adverse Event Prediction Using Machine Learning

Abstract

Adverse events are a serious issue in drug development, and many prediction methods using machine learning have been developed. The random split cross-validation is the de facto standard for model building and evaluation in machine learning, but care should be taken in adverse event prediction because this approach does not strictly match the real-world situation. The time split, which uses the time axis, is considered suitable for real-world prediction. However, the differences in model performance obtained using the time and random splits are not clear due to the lack of comparable studies. To understand the differences, we compared the model performance between the time and random splits using nine types of compound information as input, eight adverse events as targets, and six machine learning algorithms. The random split showed higher area under the curve values than did the time split for six of eight targets. The chemical spaces of the training and test datasets of the time split were similar, suggesting that the concept of applicability domain is insufficient to explain the differences derived from the splitting. The area under the curve differences were smaller for the protein interaction than for the other datasets. Subsequent detailed analyses suggested the danger of confounding in the use of knowledge-based information in the time split. These findings indicate the importance of understanding the differences between the time and random splits in adverse event prediction and suggest that appropriate use of the splitting strategies and interpretation of results are necessary for the real-world prediction of adverse events. We provide the analysis code and datasets used in the present study at .

Averaging Strategy for Interpretable Machine Learning on Small Datasets to Understand Element Uptake after Seed Nanotreatment

“…Therefore, relying on a single dataset split to create a model may pose challenges to the model’s representativeness and reliability. The performance of models established on small datasets would be obviously impacted by dataset size, split ratio, and split strategy, − and machine learning may not capture the full range of features and patterns present in the given training set. This underscores the importance of evaluating model performance and interpreting what a model learned across various splits.…”

Adaptability of AI for safety evaluation in regulatory science: A case study of drug-induced liver injury

“…When it comes to using adaptive models and assessing their adaptive behavior there are a number of strategies and approaches being used across the field of AI (Groce et al, 2002 ; Yang et al, 2005 ; Xiao et al, 2016 ; López and Tucker, 2018 ). Currently, a random split cross-validation model is considered the ML standard for model building and evaluation (Morita et al, 2022 ). Random split cross-validation is often found to be overoptimistic in comparison to real-world situations, while a time-split approach is considered suitable for real-world prediction (Morita et al, 2022 ).…”

“…Currently, a random split cross-validation model is considered the ML standard for model building and evaluation (Morita et al, 2022 ). Random split cross-validation is often found to be overoptimistic in comparison to real-world situations, while a time-split approach is considered suitable for real-world prediction (Morita et al, 2022 ). In this study, we proposed a time-split adaptability framework approach to exploring the adaptive behavior of an AI-based solution for drug toxicity and risk assessments within regulatory science.…”