The utility of multivariate outlier detection techniques for data quality evaluation in large studies: an application within the ONDRI project

Abstract

Background Large and complex studies are now routine, and quality assurance and quality control (QC) procedures ensure reliable results and conclusions. Standard procedures may comprise manual verification and double entry, but these labour-intensive methods often leave errors undetected. Outlier detection uses a data-driven approach to identify patterns exhibited by the majority of the data and highlights data points that deviate from these patterns. Univariate methods consider each variable independently, so observations that appear odd only when two or more variables are considered simultaneously remain undetected. We propose a data quality evaluation process that emphasizes the use of multivariate outlier detection for identifying errors, and show that univariate approaches alone are insufficient. Further, we establish an iterative process that uses multiple multivariate approaches, communication between teams, and visualization for other large-scale projects to follow. Methods We illustrate this process with preliminary neuropsychology and gait data for the vascular cognitive impairment cohort from the Ontario Neurodegenerative Disease Research Initiative, a multi-cohort observational study that aims to characterize biomarkers within and between five neurodegenerative diseases. Each dataset was evaluated four times: with and without covariate adjustment using two validated multivariate methods – Minimum Covariance Determinant (MCD) and Candès’ Robust Principal Component Analysis (RPCA) – and results were assessed in relation to two univariate methods. Outlying participants identified by multiple multivariate analyses were compiled and communicated to the data teams for verification. Results Of 161 and 148 participants in the neuropsychology and gait data, 44 and 43 were flagged by one or both multivariate methods and errors were identified for 8 and 5 participants, respectively. MCD identified all participants with errors, while RPCA identified 6/8 and 3/5 for the neuropsychology and gait data, respectively. Both outperformed univariate approaches. Adjusting for covariates had a minor effect on the participants identified as outliers, though did affect error detection. Conclusions Manual QC procedures are insufficient for large studies as many errors remain undetected. In these data, the MCD outperforms the RPCA for identifying errors, and both are more successful than univariate approaches. Therefore, data-driven multivariate outlier techniques are essential tools for QC as data become more complex. Electronic supplementary material The online version of this article (10.1186/s12874-019-0737-5) contains supplementary material, which is available to authorized users.

Ontario Neurodegenerative Disease Research Initiative (ONDRI): Structural MRI methods & outcome measures

“…In this paper, we presented the neuroimaging pipeline methods implemented in ONDRI that were used to overcome many of these challenges. To further ensure a high level of data quality, the volumetric data generated by the ONDRI structural neuroimaging team were further subjected to comprehensive quality control analysis pipelines including a novel multivariate outlier detection algorithm developed by the ONDRI neuroinformatics group for identification of anomalous observations Sunderland et al, 2019). Future work will include generating longitudinal measures that will also be made publicly available.…”

Methods for Improving Screening for Vascular Cognitive Impairment Using the Montreal Cognitive Assessment

“…17 All ONDRI data underwent a data evaluation procedure using multivariate outlier detection to identify anomalous observations, providing guidance on additional areas of possible error. 18 All participants provided informed consent and met extensive eligibility criteria for the larger ONDRI study. 16 Participants in the CVD cohort were post-acute ≥ 3 months and also met the following inclusion criteria: (a) proficient in speaking and understanding English, with self-ratings of 7 or more (corresponding to "good") for both speaking and understanding English on the Language Experience and Proficiency Questionnaire, 19 (b) 8 or more years of formal education, (c) post-acute ischemic stroke or silent stroke that was documented on MRI or CT, (d) mildmoderate stroke severity defined by scores of 0-3 on the modified Rankin Scale 1 (MRS), 20 and (e) a MoCA score of at least 18.…”

Cortical thickness estimation in individuals with cerebral small vessel disease, focal atrophy, and chronic stroke lesions

“…ONDRI's structural image processing pipeline (40) will be considered as the pre-processing step for the Corrected FS procedure. Briefly, ONDRI's neuroimaging platform used previously published and validated methods, where outputs were further subjected to comprehensive quality control measures from ON-DRI's neuroinformatic platform using a novel outlier detection algorithm for the identification of anomalous data (41,42). This comprehensive multi-feature segmentation pipeline was applied to co-registered T1, PD, T2, and FLAIR images to generate skull stripped and tissue segmentation masks for each individual, which included manual tracing of cortico-subcortical stroke lesions that were identified and verified on T1 and FLAIR images by an expert research neuroradiologist.…”