Exploiting heterogeneity in operational neural networks by synaptic plasticity

Abstract

The recently proposed network model, Operational Neural Networks (ONNs), can generalize the conventional Convolutional Neural Networks (CNNs) that are homogenous only with a linear neuron model. As a heterogenous network model, ONNs are based on a generalized neuron model that can encapsulate any set of non-linear operators to boost diversity and to learn highly complex and multi-modal functions or spaces with minimal network complexity and training data. However, the default search method to find optimal operators in ONNs, the so-called Greedy Iterative Search (GIS) method, usually takes several training sessions to find a single operator set per layer. This is not only computationally demanding, also the network heterogeneity is limited since the same set of operators will then be used for all neurons in each layer. To address this deficiency and exploit a superior level of heterogeneity, in this study the focus is drawn on searching the best-possible operator set(s) for the hidden neurons of the network based on the “Synaptic Plasticity” paradigm that poses the essential learning theory in biological neurons. During training, each operator set in the library can be evaluated by their synaptic plasticity level, ranked from the worst to the best, and an “elite” ONN can then be configured using the top-ranked operator sets found at each hidden layer. Experimental results over highly challenging problems demonstrate that the elite ONNs even with few neurons and layers can achieve a superior learning performance than GIS-based ONNs and as a result, the performance gap over the CNNs further widens.

Operational Neural Networks for Parameter-Efficient Hyperspectral Single-Image Super-Resolution

“…GOPs have demonstrated a superior diversity, as encountered in biological neural networks, which resulted in an elegant performance level on numerous challenging problems where conventional MLPs entirely failed. Following in the GOP's footsteps, Operational Neural Networks (ONNs) [8], [50], [9] were developed as a superset of CNNs. ONNs not only outperform CNNs significantly, but they are also able to learn certain problems where CNNs fail entirely.…”

Brain Tumor Segmentation and Classification from Sensor-Based Portable Microwave Brain Imaging System Using Lightweight Deep Learning Models

“…The ONN is a diverse network that has demonstrated a promising performance in a number of applications, including image denoising and image restoration. It usages a permanent set of non-linear operators to discover complicated patterns from any input [ 58 , 59 ]. On the other hand, the fixed set of operator libraries restricts ONNs ability to learn.…”

Dynamics in a light-sensitive neuron with two capacitive variables

“…Two recent works [32] claimed that the mode selection has close relation to energy level of the neuron, and then the shift in the energy level accounts for the mode selection in neural activities. When more neurons are clustered in the same region, continuous energy collection [33][34][35][36][37] or release will develop heterogeneity or defects in the local area of the neural network, and energy diversity enables parameter shift for keeping desynchronization between neurons [38][39][40].…”