Max-Pooling Dropout for Regularization of Convolutional Neural Networks

Abstract

Abstract.Recently, dropout has seen increasing use in deep learning. For deep convolutional neural networks, dropout is known to work well in fully-connected layers. However, its effect in pooling layers is still not clear. This paper demonstrates that max-pooling dropout is equivalent to randomly picking activation based on a multinomial distribution at training time. In light of this insight, we advocate employing our proposed probabilistic weighted pooling, instead of commonly used max-pooling, to act as model averaging at test time. Empirical evidence validates the superiority of probabilistic weighted pooling. We also compare max-pooling dropout and stochastic pooling, both of which introduce stochasticity based on multinomial distributions at pooling stage. Keywords: Deep learning, convolutional neural network, max-pooling dropout IntroductionDeep convolutional neural networks (CNNs) have recently been substantially improving on the state of art in computer vision. A standard CNN consists of alternating convolutional and pooling layers, with fully-connected layers on top. Compared to regular feed-forward networks with similarly-sized layers, CNNs have much fewer connections and parameters due to the local-connectivity and shared-filter architecture in convolutional layers, so they are far less prone to over-fitting. Another nice property of CNNs is that pooling operation provides a form of translation invariance and thus benefits generalization. Despite these attractive qualities and despite the fact that CNNs are much easier to train than other regular, deep, feed-forward neural networks, big CNNs with millions or billions of parameters still easily overfit relatively small training data. Dropout [1] is a recently proposed regularizer to fight against over-fitting. It is a regularization method that stochastically sets to zero the activations of hidden units for each training case at training time. This breaks up co-adaptions of feature detectors since the dropped-out units cannot influence other retained units. Another way to interpret dropout is that it yields a very efficient form of model averaging where the number of trained models is exponential in that of units, and these models share the same parameters. Dropout has also inspired other stochastic model averaging methods such as stochastic pooling [4], drop-connect [5] and maxout networks [3].Although dropout is known to work well in fully-connected layers of convolutional

Deep Convolutional Neural Network-Based Approaches for Face Recognition

“…In [31], a comprehensive theoretical analysis of the max pooling and average pooling is generated, whereas in [32] it has shown that max pooling can result in faster convergence of information, and the network picks the high-ranking features in the image thus enhancing generalization. Also, pooling layer possesses other variations such as stochastic pooling [33], spatial pyramid pooling [34], and def-pooling [35] that serves marked purposes.…”

Deep CNN Sparse Coding for Real Time Inhaler Sounds Classification

“…The CNN architecture, presented in Figure 1, consists of three convolutional layers with a max-pooling layer, a dropout function [33] and four fully connected layers. Using this structure, five different CNNs were developed as presented in Table 1.…”

A Novel Classification Method for Flutter Signals Based on the CNN and STFT

“…The Adam optimizer is utilized to update weights and bias in the entire network. Wu and Gu [18] proved that a dropout layer improves the neural network and can effectively prevent overfitting, so we added a dropout layer behind the two fully connected layers behind the network. Partial improvement of the basic LeNet-5 by the above method is effective for classifying flutter signals (Figure 1).…”