Deep Learning-Based Mapping of Tumor Infiltrating Lymphocytes in Whole Slide Images of 23 Types of Cancer

Abstract

The role of tumor infiltrating lymphocytes (TILs) as a biomarker to predict disease progression and clinical outcomes has generated tremendous interest in translational cancer research. We present an updated and enhanced deep learning workflow to classify 50x50 um tiled image patches (100x100 pixels at 20x magnification) as TIL positive or negative based on the presence of 2 or more TILs in gigapixel whole slide images (WSIs) from the Cancer Genome Atlas (TCGA). This workflow generates TIL maps to study the abundance and spatial distribution of TILs in 23 different types of cancer. We trained three state-of-the-art, popular convolutional neural network (CNN) architectures (namely VGG16, Inception-V4, and ResNet-34) with a large volume of training data, which combined manual annotations from pathologists (strong annotations) and computer-generated labels from our previously reported first-generation TIL model for 13 cancer types (model-generated annotations). Specifically, this training dataset contains TIL positive and negative patches from cancers in additional organ sites and curated data to help improve algorithmic performance by decreasing known false positives and false negatives. Our new TIL workflow also incorporates automated thresholding to convert model predictions into binary classifications to generate TIL maps. The new TIL models all achieve better performance with improvements of up to 13% in accuracy and 15% in F-score. We report these new TIL models and a curated dataset of TIL maps, referred to as TIL-Maps-23, for 7983 WSIs spanning 23 types of cancer with complex and diverse visual appearances, which will be publicly available along with the code to evaluate performance.Code Available at:https://github.com/ShahiraAbousamra/til_classification.

Quantifying Spatial Heterogeneity of Tumor-Infiltrating Lymphocytes to Predict Survival of Individual Cancer Patients

“…The analysis was carried out on the spatial maps of lymphocytic infiltrates generated from the images of hematoxylin and eosin-stained tissue slides from The Cancer Genome Atlas (TCGA) in a previous study [ 12 ]. The TIL maps were obtained by the use of convolutional neural networks on images divided into small patches (50 × 50) and trained for tumor-infiltrating lymphocyte detection [ 12 ].…”

“…The analysis was carried out on the spatial maps of lymphocytic infiltrates generated from the images of hematoxylin and eosin-stained tissue slides from The Cancer Genome Atlas (TCGA) in a previous study [ 12 ]. The TIL maps were obtained by the use of convolutional neural networks on images divided into small patches (50 × 50) and trained for tumor-infiltrating lymphocyte detection [ 12 ]. In the original paper [ 12 ], the information about TIL was stored using two TIL map scales: (i) binary (if there are TILs on the patch or not); (ii) probability (values from 0.5 to 1; closer to 1 means a higher chance of TILs; values smaller than 0.5 means that there are no TILs on the patch).…”

“…In this work, we collected and evaluated 14 different measures of spatial heterogeneity (termed spatial measures), including the one previously applied to TIL-map analysis, and also methods developed for the analysis of imaging mass spectrometry or spatial transcriptomics data. We tested these measures on TIL maps from 23 cancer types created elsewhere [ 12 ] with the ultimate goal to find the most robust and efficient indices that could be used for cancer prognosis.…”

Federated Learning for the Classification of Tumor Infiltrating Lymphocytes

“…The Stony Brook Biomedical Informatics group has actively contributed to this field of work for many years by characterizing the performance of DL pathology algorithms. Specifically, we have previously developed a DL based method to analyze WSIs to quantify distributions of TILs [23,24]. This method partitions a WSI into a regular mesh of image patches.…”

“…The method trains a classification model (based on a VGG16 pre-trained on Ima-geNet) to predict if a given image patch is TIL-positive (i.e., the patch contains 2 or more lymphocytes) or TIL-negative. This classification model has been trained and evaluated with a set of TIL-positive and TIL-negative patches from multiple cancer types with comprehensive analyses carried out to characterize performance of this method [24].…”