NL-CNN: A Resources-Constrained Deep Learning Model based on Nonlinear Convolution

Abstract

A novel convolution neural network model, abbreviated NL-CNN is proposed, where nonlinear convolution is emulated in a cascade of convolution + nonlinearity layers. The code for its implementation and some trained models are made publicly available. Performance evaluation for several widely known datasets is provided, showing several relevant features: i) for small / medium input image sizes the proposed network gives very good testing accuracy, given a low implementation complexity and model size; ii) compares favorably with other widely known resources-constrained models, for instance in comparison to MobileNetv2 provides better accuracy with several times less training times and up to ten times less parameters (memory occupied by the model); iii) has a relevant set of hyper-parameters which can be easily and rapidly tuned due to the fast training specific to it. All these features make NL-CNN suitable for IoT, smart sensing, biomedical portable instrumentation and other applications where artificial intelligence must be deployed in energy-constrained environments.

Fast Training of Light Binary Convolutional Neural Networks using Chainer and Cupy

“…1. It is a particular case of the NL-CNN model described in [19] where the nonlinearity index nl= (1,1). However, here the implementation is considerably different exploiting the deep-learning framework Chainer [14].…”

“…2. [19] and a function to generate randomly the binary kernels kers. Cell 5 is used to initialize the loop process where optimal binary kernels bk are sought by running Cell 6 multiple times and keeping evidence of the best accuracy performance.…”

“…Cell 5 is used to initialize the loop process where optimal binary kernels bk are sought by running Cell 6 multiple times and keeping evidence of the best accuracy performance. The structure of the model is defined in Cell 6 (conv_layersconvolution layers, nf1 nf2 nf3number of filters per each convolution layer and nonlinearity index [19], nl=(1,1) in most experiments) and each running of it reveals dynamic and structural performance as well (ELM training time, number of bits for parameters, etc.). To explore various random kernels, the variable use_stored=0 and Cell 6 runs as many times (trials) as needed to observe an increased performance.…”

“…In the next, several widely known datasets will be considered and indicate the best solution, giving a comparison between several solutions in terms of accuracy (on the test set), and number of bits for the parameters (considering that the output Adaline layer has weights represented as 8 bits integers). The neural model implemented is the NL-CNN [19] with nl=(1,1) nonlinearity index but with binary weights in convolutional layers and depth-wise convolutions. Unlike in the case of the fully-trainable NL-CNN our experiments show that nl=(1,1) i.e.…”