Empirical methods for the validation of Time-To-Event mathematical models taking into account uncertainty and variability: Application to EGFR+ Lung Adenocarcinoma

Abstract

Over the past several decades, metrics have been defined to assess the quality of various types of models and to compare their performance depending on their capacity to explain the variance found in real-life data. However, available validation methods are mostly designed for statistical regressions rather than for mechanistic models. To our knowledge, in the latter case, there are no consensus standards, for instance for the validation of predictions against real-world data given the variability and uncertainty of the data. In this work, we focus on the prediction of time-to-event curves using as an application example a mechanistic model of non-small cell lung cancer. We designed four empirical methods to assess both model performance and reliability of predictions: two methods based on bootstrapped versions of parametric statistical tests: log-rank and combined weighted log-ranks (MaxCombo); and two methods based on bootstrapped prediction intervals, referred to here as raw coverage and the juncture metric. We also introduced the notion of observation time uncertainty to take into consideration the real life delay between the moment when an event happens, and the moment when it is observed and reported. We highlight the advantages and disadvantages of these methods according to their application context. With this work, we stress the importance of making judicious choices for a metric with the objective of validating a given model and its predictions within a specific context of use. We also show how the reliability of the results depends both on the metric and on the statistical comparisons, and that the conditions of application and the type of available information need to be taken into account to choose the best validation strategy.

Knowledge-based mechanistic modeling accurately predicts disease progression with gefinitib in EGFR-mutant lung adenocarcinoma

“…The extraction of the list of time-to-events (for both PFS and OS) was realized using R package digitize (Wei and Royston (2017)), using the input survival times from graph reading; and the reported number at risk. TTP was inferred based on the clinical trial PFS and OS, as detailed in Jacob et al (2022). Therefore, the NEJ002 TTP dataset was deduced from the lists of time-to-events corresponding to the PFS and OS of the Maemondo/NEJ002 trial.…”

“…These are reported in Table 3: To be able to compare the ISELA model TTP to the LUX-LUNG7 dataset, the disease progression endpoint was similarly derived from clinical PFS and OS, as explained in Jacob et al (2022) and previously detailed in the calibration context. Therefore, the Lux-Lung 7 TTP dataset was deduced from the lists of times-to-event corresponding to the PFS and OS of the Lux-Lung 7 trial.…”

“…The virtual bootstrapped population size used for each test was equal to the size of the related real population. We computed 5,000 LR tests as advised in Jacob et al (2022) and added 2,000 tests to ensure convergence. If the percentage of statistically non-significant bootstrapped tests (p-value > 0.05) is greater than a given threshold, the model is considered able to reproduce the observed results.…”

“…Based on the recommendations from EMA (EMA ( 2018 Additionally to the four main steps, we propose a fifth one (5), namely the validation of the model Jacob et al (2022), in order to assess model credibility by challenging its predictability in reproducing real-world data that were not used to build the model nor to calibrate it.…”