Variance Counterbalancing for Stochastic Large-scale Learning

Abstract

Stochastic Gradient Descent (SGD) is perhaps the most frequently used method for large scale training. A common example is training a neural network over a large data set, which amounts to minimizing the corresponding mean squared error (MSE). Since the convergence of SGD is rather slow, acceleration techniques based on the notion of “Mini-Batches” have been developed. All of them however, mimicking SGD, impose diminishing step-sizes as a means to inhibit large variations in the MSE objective. In this article, we introduce random sets of mini-batches instead of individual mini-batches. We employ an objective function that minimizes the average MSE and its variance over these sets, eliminating so the need for the systematic step size reduction. This approach permits the use of state-of-the-art optimization methods, far more efficient than the gradient descent, and yields a significant performance enhancement.

Automated Glaucoma Detection Using Deep Convolutional Neural Networks

“…Optimizer: It is an algorithm that helps to minimize the loss function during training. In our study, we have used the Adam optimizer as it helps the model to converge faster without getting stuck in a suboptimal solution [34]. 5.…”