Integrating deep and radiomics features in cancer bioimaging

Abstract

Almost every clinical specialty will use artificial intelligence in the future. The first area of practical impact is expected to be the rapid and accurate interpretation of image streams such as radiology scans, histo-pathology slides, ophthalmic imaging, and any other bioimaging diagnostic systems, enriched by clinical phenotypes used as outcome labels or additional descriptors. In this study, we introduce a machine learning framework for automatic image interpretation that combines the current pattern recognition approach ("radiomics") with Deep Learning (DL). As a first application in cancer bioimaging, we apply the framework for prognosis of locoregional recurrence in head and neck squamous cell carcinoma (N=298) from Computed Tomography (CT) and Positron Emission Tomography (PET) imaging. The DL architecture is composed of two parallel cascades of Convolutional Neural Network (CNN) layers merging in a softmax classification layer. The network is first pretrained on head and neck tumor stage diagnosis, then fine-tuned on the prognostic task by internal transfer learning. In parallel, radiomics features (e.g., shape of the tumor mass, texture and pixels intensity statistics) are derived by pre-defined feature extractors on the CT/PET pairs. We compare and mix deep learning and radiomics features into a unifying classification pipeline (RADLER), where model selection and evaluation are based on a data analysis plan developed in the MAQC initiative for reproducible biomarkers. On the multimodal CT/PET cancer dataset, the mixed deep learning/radiomics approach is more accurate than using only one feature type, or image mode. Further, RADLER significantly improves over published results on the same data.

Deep Neural Networks and Transfer Learning on a Multivariate Physiological Signal Dataset

“…With a specific focus on multivariate physiological signals, the DNN approach adopted in this study can be easily adapted to simultaneously process multiple types of signals: convolutional branches can be used to extract a set of features from each type of signal that are then merged by the FCH [19].…”

“…DNNs based on Convolutional Neural Networks (CNN) are currently the state-ofthe-art models in several image classification applications that adopt an "end-to-end" paradigm: i.e., images are directly processed by the CNN without prior processing (e.g., feature extraction). The adoption of DNN and CNN in applications based on medical data (bio images and physiological signals) is rapidly growing, with a wide range of applications [19][20][21][22][23].…”

Integrative Network Fusion: A Multi-Omics Approach in Molecular Profiling

“…In its current version, the INF framework supports the integration of two or more one-dimensional omics layers. As part of our future effort we will add support for the integration of medical imaging layers, for example leveraging the extraction of histopathological features from whole slide images by deep learning (10) or using radiomics or deep features from radiological images (11). In both cases, further issues will emerge from the interactions between the omics and the non-omics data, needing particular care in the integration (12).…”

Machine Learning for Clinical Decision-Making: Challenges and Opportunities

“…shape, texture, luminosity, etc.) from images, DL can positively impact diagnosis performance (Bizzego et al, 2019;Lao et al, 2017). ML can also help discovering previously unidentified relevant features for diagnosis (Sanchez-Martinez et al, 2018).…”