Detail-Preserving Pooling in Deep Networks

Abstract

Most convolutional neural networks use some method for gradually downscaling the size of the hidden layers. This is commonly referred to as pooling, and is applied to reduce the number of parameters, improve invariance to certain distortions, and increase the receptive field size. Since pooling by nature is a lossy process, it is crucial that each such layer maintains the portion of the activations that is most important for the network's discriminability. Yet, simple maximization or averaging over blocks, max or average pooling, or plain downsampling in the form of strided convolutions are the standard. In this paper, we aim to leverage recent results on image downscaling for the purposes of deep learning. Inspired by the human visual system, which focuses on local spatial changes, we propose detailpreserving pooling (DPP), an adaptive pooling method that magnifies spatial changes and preserves important structural detail. Importantly, its parameters can be learned jointly with the rest of the network. We analyze some of its theoretical properties and show its empirical benefits on several datasets and networks, where DPP consistently outperforms previous pooling approaches.

Pixel-Adaptive Convolutional Neural Networks

“…Detail Preserving Pooling [36,45] is a recently proposed pooling layer that is useful to preserve high-frequency details when performing pooling in CNNs. PAC can model the detail-preserving pooling operations by incorporating an adapting kernel that emphasizes more distinct pixels in the neighborhood, e.g.,…”

“…We observe that PAC, despite being a simple modification to standard convolution, is highly flexible and can be seen as a generalization of several widely-used filters. Specifically, we show that PAC is a generalization of spatial convolution, bilateral filtering [2,42], and pooling operations such as average pooling and detail-preserving pooling [36]. We also implement a variant of PAC that does pixel-adaptive transposed convolution (also called deconvolution) which can be used for learnable guided upsampling of intermediate CNN representations.…”

LIP: Local Importance-Based Pooling

“…First, the prior knowledge that the maximum activation stands for the most discriminative detail, may not be always true. Second, the max operator over sliding windows hinders gradient-based optimization since in the backpropagation gradients are assigned only to the local maximums, as discussed in [33]. These sparse gradients would further enhance this inconsistence, in sense that discriminative activations will never become maximums unless current maximums are suppressed.…”

“…Detail-preserving pooling. Recent proposed detailpreserving pooling (DPP) [33] uses the detail criterion as importance function F , which is measured by the deviations of features from the activation statistics in sliding windows. DPP solves the problem of max pooling by designing more sophisticated importance function and ensuring the continuity for better gradient optimization.…”

“…For instance, LIP enables the network to preserve features of tiny targets while discarding false activations of the background clutter when recognizing or detecting small objects. Moreover, LIP is a more generic pooling method than the existing methods, in sense that it is capable of mimicking the behavior of average pooling, max pooling and detail-preserving pooling [33]. Experiments show LIP outperforms baseline methods by a large margin on ImageNet [8] with different architectures.…”