Optimization of Deep Learning Model Using an Improved Three-term Conjugate Gradient Algorithm

Abstract

The aim of this study is to present a new type of optimization algorithm to train deep learning model. In other to achieve this, we used a convex combination method to combine the coefficients of Fletcher-Reeves (FR) and Polak-Ribiere-Polyak (PRP) on a three-term conjugate gradient method. This algorithm called Three-term PRP-FR Algorithm was implemented from scratch using python programming language alongside some existing optimizers. These optimizers were evaluated and compared based on the convergence of these optimizers using the accuracy results on the popularly known digit recognition dataset (MNIST dataset).