Challenges of Deep Learning Methods for COVID-19 Detection Using Public Datasets

Abstract

A large number of studies in the past months have proposed deep learning-based Artificial Intelligence (AI) tools for automated detection of COVID-19 using publicly available datasets of Chest X-rays (CXRs) or CT scans for training and evaluation. Most of these studies report high accuracy when classifying COVID-19 patients from normal or other commonly occurring pneumonia cases. However, these results are often obtained on cross-validation studies without an independent test set coming from a separate dataset and have biases such as the two classes to be predicted come from two completely different datasets. In this work, we investigate potential overfitting and biases in such studies by designing different experimental setups within the available public data constraints and highlight the challenges and limitations of developing deep learning models with such datasets. We propose a deep learning architecture for COVID-19 classification that combines two very popular classification networks, ResNet and Xception, and use it to carry out the experiments to investigate challenges and limitations. The results show that the deep learning models can overestimate their performance due to biases in the experimental design and overfitting to the training dataset. We compare the proposed architecture to state-of-the-art methods utilizing an independent test set for evaluation, where some of the identified bias and overfitting issues are reduced. Although our proposed deep learning architecture gives the best performance with our best possible setup, we highlight the challenges in comparing and interpreting various deep learning algorithms’ results. While the deep learning-based methods using chest imaging data show promise in being helpful for clinical management and triage of COVID-19 patients, our experiments suggest that a larger, more comprehensive database with less bias is necessary for developing tools applicable in real clinical settings.

COVID-19 identification from volumetric chest CT scans using a progressively resized 3D-CNN incorporating segmentation, augmentation, and class-rebalancing

“…An ensemble of three lightweight pre-trained SqueezeNet, ShuffleNet [116], and EfficientNet-B0 [95] at various depths and consolidates feature maps in diverse abstraction levels [ [8,38,53,67,74,84,88,102]. This brings inherent bias on the algorithms as the model tends to learn the distribution of the data source for binary classification problem [32]. Therefore, these models perform very low when used in practical settings, where the models have to adapt to data from different domains [32].…”

“…This brings inherent bias on the algorithms as the model tends to learn the distribution of the data source for binary classification problem [32]. Therefore, these models perform very low when used in practical settings, where the models have to adapt to data from different domains [32]. Recently, Morozov et al [66] launched a public chest volumetric CT scan dataset with 1110 COVID-19 related studies (see details in subsection 2.1).…”

An interpretable multi-task system for clinically applicable COVID-19 diagnosis using CXR

“…First, in most investigations, results are obtained using crossvalidation without utilizing an independent test set coming from a separate dataset that may have biases. As a result, deep learning models trained in this way are likely to overfit the distribution of training data when independent test sets are not used and are also prone to learn dataset-specific artifacts rather than the truly generalizable disease characteristics [14]. Second, most studies use only a small dataset with very few COVID-19 samples.…”

COVID-19 identification from volumetric chest CT scans using a progressively resized 3D-CNN incorporating segmentation, augmentation, and class-rebalancing

“…Though the results obtained in the current articles are promising, they exhibit limited scope for use as a CAD tool, as most of the works, especially on x-ray images, have been based on data coming from different sources for two distinct classes (Covid Vs. Normal) [14,75,76,79,[84][85][86][87]. This brings inherent bias on the algorithms as the model tends to learn the distribution of the data source for binary classification problems [74]. Therefore, these models perform very low when used in practical settings, where the models have to adapt to data from different domains [74].…”

“…This brings inherent bias on the algorithms as the model tends to learn the distribution of the data source for binary classification problems [74]. Therefore, these models perform very low when used in practical settings, where the models have to adapt to data from different domains [74]. Recently, the authors in [88] launched a public chest volumetric CT scan dataset with 1110 COVID-19 related studies (see details in Section 2.1).…”