A theoretical comparison of batch-mode, on-line, cyclic, and almost-cyclic learning

Abstract

We study and compare different neural network learning strategies: batch-mode learning, online learning, cyclic learning, and almost-cyclic learning. Incremental learning strategies require less storage capacity than batch-mode learning. However, due to the arbitrariness in the presentation order of the training patterns, incremental learning is a stochastic process; whereas batch-mode learning is deterministic. In zeroth order, i.e., as the learning parameter eta tends to zero, all learning strategies approximate the same ordinary differential equation for convenience referred to as the "ideal behavior". Using stochastic methods valid for small learning parameters eta, we derive differential equations describing the evolution of the lowest-order deviations from this ideal behavior. We compute how the asymptotic misadjustment, measuring the average asymptotic distance from a stable fixed point of the ideal behavior, scales as a function of the learning parameter and the number of training patterns. Knowing the asymptotic misadjustment, we calculate the typical number of learning steps necessary to generate a weight within order epsilon of this fixed point, both with fixed and time-dependent learning parameters. We conclude that almost-cyclic learning (learning with random cycles) is a better alternative for batch-mode learning than cyclic learning (learning with a fixed cycle).

Beyond Hebb: Exclusive-OR and Biological Learning

“…The idea of averaging over errors and updating the weights on a slower time scale than sample presentation is well known from batch learning methods [20]. In those methods, errors are determined over a whole sweep through the pattern set and subsequently weights are updated synchronously.…”

A recursive algorithm for nonlinear least-squares problems

“…Batch training enables one to deal more efficiently with large data sets, by dividing the patterns into data batches of fixed length N and by applying algorithms to one batch at a time [15]. In Fig.…”

Relaxed conditions for convergence of batch BPAP for feedforward neural networks

“…The back-propagation algorithm (BPA) is a common and widely used method for artificial neural networks training [1][2][3][4]. There are two practical ways to implement it: batch learning and online learning.…”

“…The online mode of BPA is popular due to its simplicity to implement and less likely to be trapped in a local minimum, while the stochastic nature makes it difficult to establish theoretical conditions for convergence of the algorithm [2,3]. In contrast, the use of batch mode of training provides an accurate estimate of the gradient vector, convergence to a local minimum is thereby guaranteed under simple conditions [1][2][3]. However, the success of BPA needs a priori specification of a network structure.…”