The Curious Case of Convex Neural Networks

Abstract

In this paper, we investigate a constrained formulation of neural networks where the output is a convex function of the input. We show that the convexity constraints can be enforced on both fully connected and convolutional layers, making them applicable to most architectures. The convexity constraints include restricting the weights (for all but the first layer) to be non-negative and using a non-decreasing convex activation function. Albeit simple, these constraints have profound implications on the generalization abilities of the network. We draw three valuable insights: (a) Input Output Convex Networks (IOC-NN) self regularize and almost uproot the problem of overfitting; (b) Although heavily constrained, they come close to the performance of the base architectures; and (c) The ensemble of convex networks can match or outperform the non convex counterparts. We demonstrate the efficacy of the proposed idea using thorough experiments and ablation studies on MNIST, CIFAR10, and CIFAR100 datasets with three different neural network architectures. The code for this project is publicly available at: https://github.com/sarathsp1729/Convex-Networks.

Cross-structures Deep Transfer Learning through Kantorovich potentials for Lamb Waves based Structural Health Monitoring.

“…Proposed recently in [62], they have known a growing interest thanks to their original properties. They have found applications for classification problems [63], for new control strategies [64] and to approximate the convex functions space [65]. For a given input x ∈ R d , the output z L of the network with L layers of D hidden neurons is defined recursively by the following expression for i ∈ [0, 1, .., L] :…”

CDiNN -Convex Difference Neural Networks

“…In another work, [16] describe an ICNN architecture for binary classification, where the output is a two dimensional vector. Each element of this vector is a convex function of the inputs.…”

“…In particular, when biases are ignored, even identity mapping cannot be learnt by ICNN without pass-through layers. One approach to address this problem is suggested in [16], where it is proposed to use Leaky ReLU or ELU as activation function. ELU activation function is given by output = ( ⋅( −1) if < 0, and if >= 0) and Leaky ReLU is given by output = ( if < 0, and if >= 0), where is the input and is a hyperparameter.…”