Detection of atypical data in multicenter clinical trials using unsupervised statistical monitoring

Abstract

Background/Aims A risk-based approach to clinical research may include a central statistical assessment of data quality. We investigated the operating characteristics of unsupervised statistical monitoring aimed at detecting atypical data in multicenter experiments. The approach is premised on the assumption that, save for random fluctuations and natural variations, data coming from all centers should be comparable and statistically consistent. Unsupervised statistical monitoring consists of performing as many statistical tests as possible on all trial data, in order to detect centers whose data are inconsistent with data from other centers. Methods We conducted simulations using data from a large multicenter trial conducted in Japan for patients with advanced gastric cancer. The actual trial data were contaminated in computer simulations for varying percentages of centers, percentages of patients modified within each center and numbers and types of modified variables. The unsupervised statistical monitoring software was run by a blinded team on the contaminated data sets, with the purpose of detecting the centers with contaminated data. The operating characteristics (sensitivity, specificity and Youden’s J-index) were calculated for three detection methods: one using the p-values of individual statistical tests after adjustment for multiplicity, one using a summary of all p-values for a given center, called the Data Inconsistency Score, and one using both of these methods. Results The operating characteristics of the three methods were satisfactory in situations of data contamination likely to occur in practice, specifically when a single or a few centers were contaminated. As expected, the sensitivity increased for increasing proportions of patients and increasing numbers of variables contaminated. The three methods showed a specificity better than 93% in all scenarios of contamination. The method based on the Data Inconsistency Score and individual p-values adjusted for multiplicity generally had slightly higher sensitivity at the expense of a slightly lower specificity. Conclusions The use of brute force (a computer-intensive approach that generates large numbers of statistical tests) is an effective way to check data quality in multicenter clinical trials. It can provide a cost-effective complement to other data-management and monitoring techniques.

Central statistical monitoring of investigator-led clinical trials in oncology

“…Put simply, the DIS is a weighted geometric mean of the P values of all tests performed to compare center i with all other centers. In fact, the calculation of the DIS is more complex than this formula suggests, but the technical details are unimportant here [35]. Venet et al discusses other ways of combining many statistical tests to identify data issues in multicenter trials [31].…”

“…Experience from actual trials [29,31,32,36,39] as well as extensive simulation studies [35] have shown that a statistical data quality assessment based on the principles outlined above is quite effective at detecting data errors. Experience from actual trials suggests that data errors can be broadly classified as: Whilst some of these data errors are worse than others, in so far as they may have a more profound impact on the results of the trial, all of them can potentially be detected using CSM, at a far lower cost and with much higher efficiency than through labor-intensive methods such as source data verification and other on-site data reviews.…”

Fraud in Clinical Trials

“…A "data inconsistency score" (DIS) can be calculated for each center on the basis of the P-values of all statistical tests performed for this center (a row in the matrix of P-values) (Trotta et al 2019). Statisticians at the FDA have proposed two alternative ways of scoring centers, one based on a Fisher combination test and the other on a likelihood ratio test (Xu et al 2020).…”

Detection of Fraud in a Clinical Trial Using Unsupervised Statistical Monitoring

“…Such central statistical monitoring has been suggested to detect fraud and other types of data errors in clinical trials [3][4][5][6]. Several examples of data discrepancies suggestive of inappropriate training, poor understanding of the protocol, sloppiness in collecting the data, and fabrication of data or fraud have been reported in the literature [5,[7][8][9][10].…”