Boosting Stochastic Newton Descent for Bigdata large scale classification

Abstract

Efficient Bigdata classification requires low cost learning methods. A standard approach involves Stochastic Gradient Descent algorithm (SGD) for the minimization of the Hinge Loss in the primal space. Although complexity of Stochastic Gradient Descent is linear with the number of samples these method suffers from slow convergence. In order to cope with this issue, we propose here a Boosting Stochastic Newton Descent (BSND) method for minimization of any calibrated loss in the primal space. BSND approximates the inverse Hessian by the best low-rank approximation. We validate BSND by benchmarking it against several variants of the state-of-the-art SGD algorithm on the the large scale ImageNet and Higgs dataset. We provide further core optimization for fast convergence. The results on big data set: ImageNet and Higgs display that BSND improves significantly accuracy of the SGD baseline while being faster by orders of magnitude.

Efficient Implementation of Newton-Raphson Methods for Sequential Data Prediction

“…Gradient based methods are computationally more efficient compared to other families of online learning algorithms, i.e., for a sequence of M -dimensional feature vectors {x t } t≥0 , where x t ∈ Ê M , the computational complexity is only in the order of O(M ). However, their convergence rates remain significantly slow when achieving an optimal solution, since no statistics other than the gradient is used [3], [16], [19]. Even though gradient based methods suffer from this convergence issue, they are extensively used in big data applications due to such low computational demand [20].…”

“…T ECHNOLOGICAL developments in recent years have substantially increased the amount of data gathered from real life systems [1], [2], [3], [4]. There exists a significant data flow through the recently arising applications such as large-scale sensor networks, information sensing mobile devices and web based social networks [5], [6], [7].…”