A dual decoder U-Net-based model for nuclei instance segmentation in hematoxylin and eosin-stained histological images

Abstract

Even in the era of precision medicine, with various molecular tests based on omics technologies available to improve the diagnosis process, microscopic analysis of images derived from stained tissue sections remains crucial for diagnostic and treatment decisions. Among other cellular features, both nuclei number and shape provide essential diagnostic information. With the advent of digital pathology and emerging computerized methods to analyze the digitized images, nuclei detection, their instance segmentation and classification can be performed automatically. These computerized methods support human experts and allow for faster and more objective image analysis. While methods ranging from conventional image processing techniques to machine learning-based algorithms have been proposed, supervised convolutional neural network (CNN)-based techniques have delivered the best results. In this paper, we propose a CNN-based dual decoder U-Net-based model to perform nuclei instance segmentation in hematoxylin and eosin (H&E)-stained histological images. While the encoder path of the model is developed to perform standard feature extraction, the two decoder heads are designed to predict the foreground and distance maps of all nuclei. The outputs of the two decoder branches are then merged through a watershed algorithm, followed by post-processing refinements to generate the final instance segmentation results. Moreover, to additionally perform nuclei classification, we develop an independent U-Net-based model to classify the nuclei predicted by the dual decoder model. When applied to three publicly available datasets, our method achieves excellent segmentation performance, leading to average panoptic quality values of 50.8%, 51.3%, and 62.1% for the CryoNuSeg, NuInsSeg, and MoNuSAC datasets, respectively. Moreover, our model is the top-ranked method in the MoNuSAC post-challenge leaderboard.

Improving generalization capability of deep learning-based nuclei instance segmentation by non-deterministic train time and deterministic test time stain normalization

“…In our study, we used one single dataset for training and multiple datasets for testing to evaluate the generalization capability of our proposed approach (i.e., one single trained model was tested on multiple unseen images from various test datasets). The training was done on the training set from the MoNuSeg challenge [23] , as it encompasses a large variety of nuclei from different organs and has been widely used as a benchmark dataset in previous studies [11] , [31] , [58] . It includes a total of 21,623 nuclei, found in 30 ( pixels) H&E-stained image patches extracted from whole slide images from the cancer genome atlas (TCGA) repository.…”

“…This work is based on the DDU-Net [31] as the baseline nuclei instance segmentation model. This model has shown excellent performance in the nuclear segmentation task in various datasets (e.g., it achieved the first rank on the MoNuSAC post-challenge leaderboard for multi-organ nuclear segmentation and classification challenge [9] , [49] ).…”

“…Although there are many different nuclei instance segmentation algorithms based on CNNs, the state-of-the-art strategies can be broadly categorized into three different categories. There are detection-based methods such as adapted or improved Mask R-CNN [3] , [17] , ternary segmentation models such as deep contour-aware networks (DCAN) or Kumar et al method [6] , [22] and distance-based methods such as Hover-Net [11] , two-stage U-Net algorithms [33] or dual decoder U-Net-based (DDU-Net) model [31] . Additionally, these approaches can be jointly used in order to boost the segmentation performance [3] , [53] .…”

“…We proposed and developed a hybrid approach by combining the normalization and augmentation techniques in the nuclei instance segmentation workflow. We use DDU-Net [31] , one of the state-of-the-art DL-based nuclei instance segmentation models, as the baseline. For training, we incorporate non-deterministic stain normalization based on the Macenko et al method.…”

Learning Pseudo Scale Instance Maps for Cell Localization

“…Since the size of each cell in real cell images varies, it is highly unreasonable to use circles of the same size to represent all cells. In addition to manually generating pseudo-labels, some researchers [16], [17] have used instance segmentation to simultaneously perform cell localization in cell classification tasks. However, instance segmentation annotation is often expensive and does not provide labels directly related to cell localization and counting.…”