Explanation and Prediction of Clinical Data with Imbalanced Class Distribution based on Pattern Discovery and Disentanglement

Abstract

Background: Statistical data analysis, especially the advanced machine learning (ML) methods, have attracted considerable interest in clinical practices. We are looking for interpretability of the diagnostic/prognostic results that will bring confidence to doctors, patients and their relatives in therapeutics and clinical practice. When datasets are imbalanced in diagnostic categories, we notice that the ordinary ML methods might produce results overwhelmed by the majority classes diminishing prediction accuracy. Hence, we need methods that could produce explicit transparent and interpretable results in decision-making, without sacrificing accuracy, even for data with imbalanced groups. Methods: In order to interpret the clinical patterns and conduct diagnostic prediction of patients with high accuracy, we develop a novel method, Pattern Discovery and Disentanglement for Clinical Data Analysis (cPDD), which is able to discover patterns (correlated traits/indicants) and use them to classify clinical data even if the class distribution is imbalanced. In the most general setting, a relational dataset is a large table such that each column represents an attribute (trait/indicant), and each row contains a set of attribute values (AVs) of an entity (patient). Compared to the existing pattern discovery approaches, cPDD can discover a small succinct set of statistically significant high-order patterns from clinical data for interpreting and predicting the disease class of the patients even with groups small and rare.Results: Experiments on synthetic and thoracic clinical dataset showed that cPDD can 1) discover fewer patterns compared to other existing pattern discovery methods; 2) allow the users to interpret succinct sets of patterns coming from uncorrelated sources, even the groups are rare/small; and 3) obtain better performance in prediction compared to other interpretable classification approaches. Conclusions: In conclusion, cPDD discovers fewer patterns with greater comprehensive coverage to improve the interpretability of patterns discovered. Experimental results on synthetic data validated that cPDD discovers all patterns implanted in the data, displays them precisely and succinctly with statistical support for interpretation and prediction, a capability which the traditional ML methods lack. The success of cPDD as a novel interpretable method in solving the imbalanced class problem shows its great potential to clinical data analysis for years to come.

Cohort Characteristics and Factors Associated With Cannabis Use Among Adolescents in Canada Using Pattern Discovery and Disentanglement Method

“…However, frequency is difficult to measure how significant of the association. Hence, as reported in our previous paper [21],…”

“…However, frequency is difficult to measure how significant of the association. Hence, as reported in our previous paper [21], the adjusted standardized residual (AR) is more appropriate to represent the statistical weights of the AV pair than their frequency of co-occurrences. The AR for an AV pair (say.…”

“…When datasets are imbalanced in diagnostic categories, the ordinary ML methods might produce results overwhelmed by the majority classes while diminishing significant patterns in minority classes [21]. To balance the dataset, researchers usually use under-sampling and oversampling methods [22].…”

“…It discovers associations without using class information and sampling methods. It shows robust performance for discovering patterns from an imbalanced dataset [21]. No matter whether the data is balanced or imbalanced, PDD can naturally handle it robustly and steadily [21].…”

Cluster-Based Ensemble Learning Model for Rapid Detection of Aortic Dissection

“…Huang et al [6] used an enhanced resampling method of electronic medical records to classify and predict Major adverse cardiac events (MACEs) of acute coronary syndrome (ACS). Zhou et al [7] proposed an interpretable pattern discovery method from the perspective of statistical learning methods to interpret clinical chest data and make classification predictions. Song et al [8] used deep learning technology to develop a histopathological detection system for gastric cancer detection that can be used for clinical diagnosis.…”