UNETRIS: transformer-based nuclear instance segmentation for three-dimensional fluorescence microscopy images

Abstract

Automated cellular nuclei segmentation is often an important step for digital pathology and other analyses such as computer aided diagnosis. Most existing machine learning methods for microscopy image analysis require postprocessing such as watershed transform or connected component analysis to obtain instance segmentation from semantic segmentation results. This becomes prohibitively expensive computationally especially when used with 3D microscopy volumes. UNet Transformers for Instance Segmentation (UNETRIS) is proposed to eliminate the postprocessing steps necessary for nuclei instance segmentation in 3D microscopy images. UNETRIS, an extension of UNETR which utilizes a transformer as the encoder in the successful "U-shaped" network design for the encoder-decoder structure of U-Net, uses additional transformers to separate individual instances of cell nuclei directly during the inference step without the need for expensive postprocessing steps. UNETRIS does not require but can use manual ground truth annotations for training. UNETRIS was tested on a variety of microscopy volumes collected from multiple regions of organ tissues.