Enhanced Transfer Learning Through Medical Imaging and Patient Demographic Data Fusion

Abstract

In this work we examine the performance enhancement in classification of medical imaging data when image features are combined with associated non-image data. We compare the performance of eight state-of-the-art deep neural networks in classification tasks when using only image features, compared to when these are combined with patient metadata. We utilise transfer learning with networks pretrained on ImageNet used directly as feature extractors and fine tuned on the target domain. Our experiments show that performance can be significantly enhanced with the inclusion of metadata and use interpretability methods to identify which features lead to these enhancements. Furthermore, our results indicate that the performance enhancement for natural medical imaging (e.g. optical images) benefit most from direct use of pre-trained models, whereas non natural images (e.g. representations of non imaging data) benefit most from fine tuning pre-trained networks. These enhancements come at a negligible additional cost in computation time, and therefore is a practical method for other applications.

The Application of Radiomics and AI to Molecular Imaging for Prostate Cancer

“…Typically, this is used in supervised tasks, but it also enables the use of the pre-trained network as a feature extractor for clustering or classification tasks. The performance of transfer learning models demonstrates their potential impact in clinical tasks [ 66 , 67 , 68 ], such as lesion detection [ 69 ] and combining patient imaging and demographics data [ 70 ]. Pretrained models transferred from one task to another have been shown to perform at least as well as models trained (from random network weights) for the specific task, and fine-tuning these models improves robustness [ 69 ].…”