Non-linear Convolution Filters for CNN-Based Learning

Abstract

During the last years, Convolutional Neural Networks (CNNs) have achieved state-of-the-art performance in image classification. Their architectures have largely drawn inspiration by models of the primate visual system. However, while recent research results of neuroscience prove the existence of non-linear operations in the response of complex visual cells, little effort has been devoted to extend the convolution technique to non-linear forms. Typical convolutional layers are linear systems, hence their expressiveness is limited. To overcome this, various non-linearities have been used as activation functions inside CNNs, while also many pooling strategies have been applied. We address the issue of developing a convolution method in the context of a computational model of the visual cortex, exploring quadratic forms through the Volterra kernels. Such forms, constituting a more rich function space, are used as approximations of the response profile of visual cells. Our proposed second-order convolution is tested on CIFAR-10 and CIFAR-100. We show that a network which combines linear and non-linear filters in its convolutional layers, can outperform networks that use standard linear filters with the same architecture, yielding results competitive with the state-of-the-art on these datasets.

Enhanced robustness of convolutional networks with a push–pull inhibition layer

“…A layer of nonlinear convolutions inside CNNs was proposed in [50]. The authors were inspired by studies of nonlinear processes in early stages of the visual system, and modeled them by means of Volterra convolutions.…”

Kervolutional Neural Networks

“…In recent years, researchers have been paying much attentions on the extension of convolution. To enable the expressibility of convolution for complex cells, the non-linear convolutional network [57] extends convolution to non-linear space by directly introducing high order terms. However, as indicated before, this introduces a large number additional parameters and increases the training complexity exponentially.…”

“…However, the convolutional layers are linear and designed to mimic the behavior of simple cells in human visual cortex [57], hence they are not able to express the non-linear behaviors of the complex and hypercomplex cells inside the striate cortex. It was also demonstrated that higher order non-linear feature maps are able to make subsequent linear classifiers more discriminative [37,1,9].…”

Convolutional Neural Networks with Recurrent Neural Filters

“…Linear convolution filters are dominated in CNNbased systems for both computer vision and natural language processing tasks. One exception is the work of Zoumpourlis et al (2017), which proposes a convolution filter that is a function with quadratic forms through the Volterra kernels. However, this non-linear convolution filter is developed in the context of a computational model of the visual cortex, which is not suitable for NLP problems.…”