SegBlocks: Towards Block-Based Adaptive Resolution Networks for Fast Segmentation

Abstract

SegBlocks reduces the computational cost of existing neural networks, by dynamically adjusting the processing resolution of image regions based on their complexity. Our method splits an image into blocks and downsamples blocks of low complexity, reducing the number of operations and memory consumption. A lightweight policy network, selecting the complex regions, is trained using reinforcement learning. In addition, we introduce several modules implemented in CUDA to process images in blocks. Most important, our novel BlockPad module prevents the feature discontinuities at block borders of which existing methods suffer, while keeping memory consumption under control. Our experiments on Cityscapes and Mapillary Vistas semantic segmentation show that dynamically processing images offers a better accuracy versus complexity trade-off compared to static baselines of similar complexity. For instance, our method reduces the number of floating-point operations of SwiftNet-RN18 by 60% and increases the inference speed by 50%, with only 0.3% decrease in mIoU accuracy on Cityscapes.

SegBlocks: Block-Based Dynamic Resolution Networks for Real-Time Segmentation

“…Module execution time analysis: We profile the time characteristics of our block modules to analyze their overhead. The heuristic policy is given in [28].…”

“…The 'random' policy randomly selects blocks for high-resolution processing, with the amount of selected blocks given by the target percentage. The heuristic policy was proposed in [28] and selects the regions with the highest visual change, based on the average L2 distance between high-and low-resolution versions of a block.…”

“…This work is an extension of a 4-page short paper [28], where block-based processing was introduced in combination with a simple heuristic-based downsampling policy that selects regions where the loss of visual detail is most significant. Here, we provide more details on the blockbased processing framework and we introduce a superior reinforcement learning downsampling policy that is trained specifically for the task at hand.…”

“…The random policy randomly selects a percentage of blocks per frame for high-resolution processing. The heuristic policy is given in[28].…”

Real-Time Semantic Image Segmentation with Deep Learning for Autonomous Driving: A Survey

“…A novel and innovative method to speed up inference time is block-based processing, where as in [17] an image is split into blocks and adjusts the resolution of each block by downsampling the less important. This reduction of the processing resolution results in the reduction of the computational burden and the memory consumption.…”