Collaborative Global-Local Networks for Memory-Efficient Segmentation of Ultra-High Resolution Images

Abstract

Mobile memory capacity (a) Best performance achievable Mobile memory capacity (b) Performance trained on global image Mobile memory capacity (c) Performance trained on local patchesFigure 1: Inference memory and mean intersection over union (mIoU) accuracy on the DeepGlobe dataset [1]. (a): Comparison of best achievable mIoU v.s. memory for different segmentation methods. (b): mIoU/memory with different global image sizes (downsampling rate shown in scale annotations). (c): mIoU/memory with different local patch sizes (normalized patch size shown in scale annotations). GLNet (red dots) integrates both global and local information in a compact way, contributing to a well-balanced trade-off between accuracy and memory usage. See Section 4 for experiment details. Methods studied: ICNet [2], DeepLabv3+ [3], FPN [4], FCN-8s [5], UNet [6], PSPNet [7], SegNet [8], and the proposed GLNet. AbstractSegmentation of ultra-high resolution images is increasingly demanded, yet poses significant challenges for algorithm efficiency, in particular considering the (GPU) memory limits. Current approaches either downsample an ultrahigh resolution image or crop it into small patches for separate processing. In either way, the loss of local fine details or global contextual information results in limited segmentation accuracy. We propose collaborative Global-Local Networks (GLNet) to effectively preserve both global and local information in a highly memory-efficient manner. GLNet is composed of a global branch and a local branch, taking the downsampled entire image and its cropped local patches as respective inputs. For segmentation, GLNet deeply fuses feature maps from two branches, capturing both the high-resolution fine structures from zoomed-in local patches and the contextual dependency from the downsampled input. To further resolve the potential class imbalance problem between background and foreground regions, we present a coarse-to-fine variant of GLNet, also being * The first two authors contributed equally. memory-efficient. Extensive experiments and analyses have been performed on three real-world ultra-high aerial and medical image datasets (resolution up to 30 million pixels). With only one single 1080Ti GPU and less than 2GB memory used, our GLNet yields high-quality segmentation results and achieves much more competitive accuracymemory usage trade-offs compared to state-of-the-arts.

Automated Segmentation of Amyloid-β Stained Whole Slide Images of Brain Tissue

“…Most of these deep learning methods have only been tested on images with low to medium resolution of up to a few megapixels. Chen et al [15] regarded images with resolution up to 30 million pixels as ultra-high resolution images. They proposed a method that integrates both global downsampled images and local patches while they only tested their methods in images with resolution up to 30 million pixels.…”

“…However, the average resolution of our WSIs exceeds 2 gigapixels. Downsampling used in [15] cannot be directly transferred to our case as we have only 30 WSIs with proper annotations, which indicates the shortage of training data, while the datasets used in their experiments consist of more than 2,000 images. Besides, we need to preserve minute details of pathologies (such as plaques) and cells in our WSIs while downsampling will unavoidably lose these important features.…”

“…To solve this resolution issue, current methods can be divided into two categories: either downsample original images of ultra-high resolution or divide original images into small patches for separate processing [15]. To provide a proof of concept for a workflow for GM/WM segmentation algorithm, we utilize 30 WSIs with annotations, far less than the dataset used in [15], thus downsampling is not applicable for deep learning based methods. Both the plaques and cells with sizes ranging from 20 × 20 to 100 × 100 in WSIs are distinguishable features of GM and WM while simple downsampling may make them less clear or even disappear [16], which results in their limited performance.…”

Foveation for Segmentation of Mega-Pixel Histology Images

“…A considerable amount of work has attempted to alleviate the issue of subjectivity in tuning this trade-off by learning to merge multi-scale information, in both medical imaging [3] and computer vision [4,5]. These works optimise model performance by exploiting the information from multi-scale sources.…”

“…Another is [6], which is designed to work specifically with histopathology images. In general computer vision, the authors of [5] boost the learning efficiency of multi-scale information by enforcing global and local streams to interactively exchange information with each other. On the other hand, Chen et al [4] perform multi-scale feature aggregation via an attention mechanism, which weights the prediction score from multi-scale parallel networks.…”