Deep Learning Based Registration of Serial Whole-slide Histopathology Images in Different Stains

Abstract

Whole-slide image (WSI) analysis has been largely performed in a 2D tissue space to support routine pathology diagnosis and imaging based biomedical research. For a more definitive representation and characterization of the tissue spatial space, it is critical to extend such tissue based investigations to a 3D space by spatially aligning 2D serial sections, which are often stained differently, such as Hematoxylin and Eosin (H\&E) and Immunohistochemistry (IHC) stains. However, registration of whole slide images is challenged by the overwhelmingly scale of images, the complexity of local histology structure changes across slides, and significant variations of tissue appearance between staining methods. We propose a novel translation based registration network CycGANRegNet using deep learning for serial WSI images in different stains, which requires no prior deformation field information for deep model training. We first generate synthetic IHC slides from H\&E slides through a robust image synthesis algorithm. The synthetic IHC images and the real IHC images are then registered through a Fully Convolutional Network with multi-scale based deformable vector fields and a joint loss optimization for enhancing image alignment. We perform the registration at original image resolution with a patch-wide approach, thus tissue details at the highest resolution are retained in the results. CycGANRegNet outperforms both the state-of-the-art conventional and deep learning-based registration methods based on the evaluation using a serial WSI image dataset in H\&E stain and IHC stain with two biomarkers from 76 breast cancer patients. The experimental and comparison results demonstrate that CycGANRegNet can produce promising registration results with serial WSIs in different stains, suggesting its potential for integrative 3D tissue-based biomedical investigations.

Assessment of an AI virtual staining model performance across same and serial tissue sections using CD3+T cell ground truth

“…To address both inter- and intra-observer discrepancies in the annotation and scoring of cell phenotypes, there has been a growing interest in the utilization of digitized cell-level data as the definitive reference for predicting cell types in situ 10 . Commonly, immunohistochemical (IHC) staining is applied to a tissue section that is consecutive to another one stained with H&E, assuming that similar cells maintain identical locations across both sections.…”