Learning to Zoom: A Saliency-Based Sampling Layer for Neural Networks

Abstract

We introduce a saliency-based distortion layer for convolutional neural networks that helps to improve the spatial sampling of input data for a given task. Our differentiable layer can be added as a preprocessing block to existing task networks and trained altogether in an end-to-end fashion. The effect of the layer is to efficiently estimate how to sample from the original data in order to boost task performance. For example, for an image classification task in which the original data might range in size up to several megapixels, but where the desired input images to the task network are much smaller, our layer learns how best to sample from the underlying high resolution data in a manner which preserves task-relevant information better than uniform downsampling. This has the effect of creating distorted, caricature-like intermediate images, in which idiosyncratic elements of the image that improve task performance are zoomed and exaggerated. Unlike alternative approaches such as spatial transformer networks, our proposed layer is inspired by image saliency, computed efficiently from uniformly downsampled data, and degrades gracefully to a uniform sampling strategy under uncertainty. We apply our layer to improve existing networks for the tasks of human gaze estimation and fine-grained object classification. Code for our method is available in: http://github.com/recasens/Saliency-Sampler.

Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-Grained Image Recognition

“…However, 448 input increases the computational cost (i.e., flops) by four times compared to 224 input. SSN [22] obtains a better results than DT-RAM [19], and our TASN can further obtain 2.9% relative improvement. Such improvements mainly come from two aspects: 1) a better sampling mechanism considering spatial distortion (1.2%), and 2) a better fine-grained details optimizing strategy (1.7%).…”

“…But without explicit guidance, it is hard to learn non-uniformed sampling parameters for sophisticated tasks such as fine-grained recognition, thus they finally learned two parts without non-uniformed sampling. SSN [22] firstly proposed to use saliency maps as the guidance of non-uniformed sampling and obtained significant improvements. Different from them, our attention sampler 1) conduct non-uniformed sampling based on trilinear attention maps, and 2) decomposes attention maps into two dimensions to reduce spatial distortion effects.…”

“…• DT-RAM [19]: Dynamic computational time model for recurrent visual attention, which attends on the most discriminative parts by dynamic steps. • SSN [22]: Saliency-based sampling networks, which conduct non-uniformed sampling based on saliency map in an end-to-end way. which is a novel self-supervision mechanism to effectively localize informative regions without the need of bounding-box/part annotations.…”

Efficient Segmentation: Learning Downsampling Near Semantic Boundaries

“…Spatial Transformer Networks [28,42] learn spatial transformations (warping) of the CNN input. They explore different parameterizations for spatial transformation including affine, projective, splines [28] or specially designed saliency-based layers [42]. Their focus is to undo different data distortions or to "zoom-in" on salient regions, while our approach is focused on efficient downsampling retaining as much information around semantic boundaries as possible.…”

Foveated Image Processing for Faster Object Detection and Recognition in Embedded Systems Using Deep Convolutional Neural Networks

“…The work in [1] has demonstrated the advantages of foveated image processing with regard to improvements in computational efficiency (but did not address CNNs). In recent models of visual saliency using CNNs, images have been applied to networks using a foveal transform [2] [8]. However, those works did not investigate image size reduction and frame-rate speed-up, which is of critical importance for embedded systems.…”