Background: Tumor programmed death-ligand 1 (PD-L1) status is useful in determining which patients may benefit from programmed death-1 (PD-1)/PD-L1 inhibitors. However, little is known about the association between PD-L1 status and tumor histopathological patterns. Using deep learning, we predicted PD-L1 status from hematoxylin and eosin (H and E) whole-slide images (WSIs) of nonsmall cell lung cancer (NSCLC) tumor samples. Materials and Methods: One hundred and thirty NSCLC patients were randomly assigned to training ( n = 48) or test ( n = 82) cohorts. A pair of H and E and PD-L1-immunostained WSIs was obtained for each patient. A pathologist annotated PD-L1 positive and negative tumor regions on the training samples using immunostained WSIs for reference. From the H and E WSIs, over 145,000 training tiles were generated and used to train a multi-field-of-view deep learning model with a residual neural network backbone. Results: The trained model accurately predicted tumor PD-L1 status on the held-out test cohort of H and E WSIs, which was balanced for PD-L1 status (area under the receiver operating characteristic curve [AUC] =0.80, P << 0.01). The model remained effective over a range of PD-L1 cutoff thresholds (AUC = 0.67–0.81, P ≤ 0.01) and when different proportions of the labels were randomly shuffled to simulate interpathologist disagreement (AUC = 0.63–0.77, P ≤ 0.03). Conclusions: A robust deep learning model was developed to predict tumor PD-L1 status from H and E WSIs in NSCLC. These results suggest that PD-L1 expression is correlated with the morphological features of the tumor microenvironment.

Patient responses to cancer immunotherapy are shaped by their unique genomic landscape and tumor microenvironment. Clinical advances in immunotherapy are changing the treatment landscape by enhancing a patient's immune response to eliminate cancer cells. While this provides potentially beneficial treatment options for many patients, only a minority of these patients respond to immunotherapy. In this work, we examined RNA-seq data and digital pathology images from individual patient tumors to more accurately characterize the tumor-immune microenvironment. Several studies implicate an inflamed microenvironment and increased percentage of tumor infiltrating immune cells with better response to specific immunotherapies in certain cancer types. We developed NEXT (Neural-based models for integrating gene EXpression and visual Texture features) to more accurately model immune infiltration in solid tumors. To demonstrate the utility of the NEXT framework, we predicted immune infiltrates across four different cancer types and evaluated our predictions against expert pathology review. Our analyses demonstrate that integration of imaging features improves prediction of the immune infiltrate. Of note, this effect was preferentially observed for B cells and CD8 T cells. In sum, our work effectively integrates both RNA-seq and imaging data in a clinical setting and provides a more reliable and accurate prediction of the immune composition in individual patient tumors.

To achieve minimum DNA input requirements for next-generation sequencing (NGS), pathologists visually estimate macrodissection and slide count decisions. Unfortunately, misestimation may cause tissue waste and increased laboratory costs. We developed an artificial intelligence (AI)-augmented smart pathology review system (SmartPath) to empower pathologists with quantitative metrics for accurately determining tissue extraction parameters. SmartPath uses two deep learning architectures, a U-Net based network for cell segmentation and a multi-field-of-view convolutional network for tumor area segmentation, to extract features from digitized H&E-stained formalin-fixed paraffin-embedded slides. From the segmented tumor area, SmartPath suggests a macrodissection area. To predict DNA yield per slide, the extracted features from within the macrodissection area are correlated with known DNA yields to fit a regularized linear model ( R = 0.85). Then, a pathologist-defined target yield divided by the predicted DNA yield per slide gives the number of slides to scrape. Following model development, an internal validation trial was conducted within the Tempus Labs molecular sequencing laboratory. We evaluated our system on 501 clinical colorectal cancer slides, where half received SmartPath-augmented review and half traditional pathologist review. The SmartPath cohort had 25% more DNA yields within a desired target range of 100–2000 ng. The number of extraction attempts was statistically unchanged between cohorts. The SmartPath system recommended fewer slides to scrape for large tissue sections, saving tissue in these cases. Conversely, SmartPath recommended more slides to scrape for samples with scant tissue sections, especially those with degraded DNA, helping prevent costly re-extraction due to insufficient extraction yield. A statistical analysis was performed to measure the impact of covariates on the results, offering insights on how to improve future applications of SmartPath. With these improvements, AI-augmented histopathologic review has the potential to decrease tissue waste, sequencing time, and laboratory costs by optimizing DNA yields, especially for samples with scant tissue and/or degraded DNA.

No abstract

Patient responses to cancer immunotherapy are shaped by their unique genomic landscape and tumor microenvironment. Clinical advances in immunotherapy are changing the treatment landscape by enhancing a patient’s immune response to eliminate cancer cells. While this provides potentially beneficial treatment options for many patients, only a minority of these patients respond to immunotherapy. In this work, we examined RNA-seq data and digital pathology images from individual patient tumors to more accurately characterize the tumor-immune microenvironment. Several studies implicate an inflamed microenvironment and increased percentage of tumor-infiltrating immune cells with better response to specific immunotherapies in certain cancer types. We developed NEXT (Neural-based models for integrating gene EXpression and visual Texture features) to more accurately model immune infiltration in solid tumors. To demonstrate the utility of the NEXT framework, we predicted immune infiltrates across four different cancer types and evaluated our predictions against expert pathology review. Our analyses demonstrate that integration of imaging features improves prediction of the immune infiltrate. Of note, this effect was preferentially observed for natural killer, macrophage, and CD8 T cells. In sum, our work effectively integrates both RNA-seq and imaging data in a clinical setting and provides a more reliable and accurate prediction of the immune composition in individual patient tumors. Citation Format: Derek Reiman, Lingdao Sha, Irvin Ho, Timothy Tan, Denise Lau, Aly A Khan. Integrating RNA expression and visual features for immune infiltrate prediction [abstract]. In: Proceedings of the AACR Special Conference on Tumor Immunology and Immunotherapy; 2018 Nov 27-30; Miami Beach, FL. Philadelphia (PA): AACR; Cancer Immunol Res 2020;8(4 Suppl):Abstract nr B57.