Follow-Up on the Use of Machine Learning in Clinical Quality Assurance: Can We Detect Adverse Event Under-Reporting in Oncology Trials?

Abstract

Dear Editor, In a previous project [1], we developed a predictive model that enabled Roche/Genentech quality leads oversight of adverse event (AE) reporting. External clinical trial datasets such as Project Data Sphere (PDS) [2] allowed us to further test our machine learning-based approach to alleviate concerns of overfitting and to demonstrate the reproducibility of our research.Our primary objective was to further validate our model for detection of AE under-reporting using PDS data. Our secondary objective was to build an oncology-specific model using a combined dataset of Roche and PDS data. The scope remained as predicting AEs-not adverse drug reactionsthat occur in clinical trials. Good clinical practice requires all AEs (regardless of the causal relationship between the drug intake and the events) to be reported in a timely manner [3].The curation process of downloadable PDS studies (as of November 2019) left five studies that fulfilled our data requirements, as sponsors are not required to share the full datasets. They were large phase III trials and included 742 investigator sites, 2363 subjects, and 51,847 visits. Hence, we could use PDS data to achieve our objectives.The oncology-specific model was built using the methodology described in our previous manuscript [1]. We used a combined dataset of 53 completed oncology studies (Roche + PDS). Our final model used 38 features built from patient and study attributes.To test whether our model can be applied to non-Roche studies, we compared the quality of the predictions using a scatter plot ( Fig. 1a) and found that, within a range of 0-150 on both axes (> 94% of all datapoints, our region of interest [ROI]), the predictions matched the observed values for both datasets equally well. To quantify the goodness of fit, we used scale-independent performance metrics (which are adequate for comparing the goodness of fit of different datasets used by the same model [4]): symmetric mean absolute percentage error (SMAPE) [5] and symmetric mean absolute poisson significance level (SMASL). The latter is calculated by subtracting 0.5 from each poisson significance level measurement, converting it to its absolute value, and taking the mean. SMASL puts equal weight on over-and under-predicting and has a range from 0 to 0.5 (i.e., The smaller the value the better the fit). Considering SMAPE, average predictions for the PDS study sites were slightly better than for the Roche study sites, whereas the reverse was true for SMASL ( Fig. 1b). We concluded that the goodness of fit for both datasets using our model was very similar within the ROI.For the secondary objective, we tested how well the oncology model (using Roche and PDS data and the same algorithm [1]) would detect simulated test cases on data not used for model training. For relevant simulation scenarios of 25%, 50%, and 75% under-reporting on the site level, our model scored an area under the curve (AUC) of the receiver operating characteristic curve of 0.60, 0.77, and 0.90, respectively. These AUC values were on...

Leveraging analytics to assure quality during the Covid-19 pandemic - The COVACTA clinical study example

“…Leveraging analytics and integrating the outputs and their interpretation in a Quality Analytics Review Report could serve as quality evidence and reduce the burden of on-site audits and inspections. As a next step, we will continue to engage with other pharmaceutical sponsors, industry associations and Health Authorities [ 8 ] to accelerate the use of advanced analytics for clinical QA [ [6] , [7] , [8] , [9] , [10] ] - the overall goal being to improve quality and compliance throughout the trial and thereby contribute to an accelerated drug approval process for the benefit of patients.…”

“…The main areas of focus with respect to patient safety were assessing the risk for Adverse Events (AE) under-reporting (including Serious Adverse Events (SAE) and Adverse Events of Special Interests (AESI)) and ensuring patients had been dosed properly with tocilizumab. Individual investigator sites have been monitored for potential AE under-reporting, using descriptive analytics, complemented by a machine-learning approach [ 6 , [7] , [10] ]. Due to the uniqueness of this study (short timelines, high mortality) the statistical models trained on past study data could not always be applied.…”

“…As COVACTA was being conducted under the circumstances of the Covid-19 pandemic that forbade the possibility of on-site monitoring and on-site investigator audits, an alternative approach to ensure the clinical quality of the study was required. Roche/Genentech Product Development Quality (PDQ) had implemented a set of data-driven solutions to complement and augment traditional QA approaches to improve the quality and oversight of GCP regulated activities [ [6] , [7] , [8] , [9] , [10] ]. These analytics had been developed since 2018 and the COVACTA study provided the opportunity to fully implement their use.…”

Using Statistical Modeling for Enhanced and Flexible Pharmacovigilance Audit Risk Assessment and Planning

“…As noted above, the model output is not the sole driver for audit selection, but rather contributes to a wider set of data points that drive the sample selection. As such, the model is best classified as Quality Decision Support Tool—part of a broader effort to use data and statistics to enhance quality assurance activities [ 8 , 9 ]. In addition, the temporal nature of the model outputs enables a quality assurance program that is more targeted toward the current/future potential compliance issues rather than a retrospective approach.…”

Harnessing the Power of Quality Assurance Data: Can We Use Statistical Modeling for Quality Risk Assessment of Clinical Trials?

“…Meaning that both features were very process-centric. For further investigation on clinical trial safety reporting, other approaches using machine learning for anomaly detection [16,17] had demonstrated value and might be more suitable.…”