Performance Comparison of CNN Models Using Gradient Flow Analysis

Abstract

Convolutional neural networks (CNNs) are widely used among the various deep learning techniques available because of their superior performance in the fields of computer vision and natural language processing. CNNs can effectively extract the locality and correlation of input data using structures in which convolutional layers are successively applied to the input data. In general, the performance of neural networks has improved as the depth of CNNs has increased. However, an increase in the depth of a CNN is not always accompanied by an increase in the accuracy of the neural network. This is because the gradient vanishing problem may arise, causing the weights of the weighted layers to fail to converge. Accordingly, the gradient flows of the VGGNet, ResNet, SENet, and DenseNet models were analyzed and compared in this study, and the reasons for the differences in the error rate performances of the models were derived.

A new approach to training neural networks using natural gradient descent with momentum based on Dirichlet distributions

“…Натуральный градиентный спуск ( [11], [13]) с импульсом, удовлетворяющий условию Нестерова, может быть представлен следующим образом: (k) ) (τ -параметр демпфирования), F -матрица Фишера, которая учитывает кривизну поверхности f для обхода локальных минимумов и отличает натуральный градиентный спуск (2) от стохастического (1). Определение матрицы Фишера берет свое начало еще с определения градиентного потока на гладких Римановых многообразиях в [8], где свойства производных (градиентов) и кривизны уже рассмотрены в общих случаях. Данный подход уже пытались использовать в методах оптимизации в [14].…”

“…Особенно остро встает вопрос нахождения минимума в машинном обучении, где процесс оптимизации функции потерь влияет на конечную точность. Для решения данной проблемы был предложен градиентный поток из [8], представляющий собой произведение метрического тензора на гладком многообразии и градиента оптимизируемой функции. Такой подход ускорил процесс минимизации функции потерь в нейронных сетях, но в данной статье будут использоваться многообразия вероятностных распределений вместо гладких.…”

A Smart Home Energy Management System Utilizing Neurocomputing-Based Time-Series Load Modeling and Forecasting Facilitated by Energy Decomposition for Smart Home Automation

“…For a model such as the fully connected feedforward DNN illustrated in Fig. 5, the rectified linear unit (ReLU)-style activation function is commonly used for the neurons in the hidden layers, rather than an S-shaped function such as the sigmoid function, in order to overcome gradient vanishing [35], [36]; in the context of DNNs, the problem of vanishing gradients can often arise for a model configured with Sshaped activation functions for the hidden neurons to be trained by a stochastic gradient-based weight optimizer such as Adam, which was first proposed in [37], for their most appropriate weight coefficients including biases.…”

Identification of Corn Leaf Diseases Comprising of Blight, Grey Spot and Rust Using DenseNet-201

“…DenseNet-201 models are built with several parallel layer skips that aid in the training of deeper network architectures to identify corn leaf diseases. DenseNet-201 encompasses a concatenate convolutional network that extricated it from other identifier algorithms, which upsurges variation in the input of subsequent layers and enriches efficiency (Noh, 2021). The layers between two adjacent blocks are implied to as transition layers or concatenate convolutional neural network layers.…”