Convex Optimization for Big Data: Scalable, randomized, and parallel algorithms for big data analytics

Abstract

This article reviews recent advances in convex optimization algorithms for Big Data, which aim to reduce the computational, storage, and communications bottlenecks. We provide an overview of this emerging field, describe contemporary approximation techniques like first-order methods and randomization for scalability, and survey the important role of parallel and distributed computation.The new Big Data algorithms are based on surprisingly simple principles and attain staggering accelerations even on classical problems. Convex optimization in the wake of Big DataConvexity in signal processing dates back to the dawn of the field, with problems like least-squares being ubiquitous across nearly all sub-areas. However, the importance of convex formulations and optimization has increased even more dramatically in the last decade due to the rise of new theory for structured sparsity and rank minimization, and successful statistical learning models like support vector machines. These formulations are now employed in a wide variety of signal processing applications including compressive sensing, medical imaging, geophysics, and bioinformatics [1-4].There are several important reasons for this explosion of interest, with two of the most obvious ones being the existence of efficient algorithms for computing globally optimal solutions and the ability to use convex geometry to prove useful properties about the solution [1,2]. A unified convex formulation also transfers useful knowledge across different disciplines, such as sampling and computation, that focus on different aspects of the same underlying mathematical problem [5].However, the renewed popularity of convex optimization places convex algorithms under tremendous pressure to accommodate increasingly large data sets and to solve problems in unprecedented dimensions. Internet, text, and imaging problems (among a myriad of other examples) no longer produce data sizes from megabytes to gigabytes, but rather from terabytes to exabytes. Despite

Big data algorithms beyond machine learning

“…It would be imprudent to claim that we have covered all such methods, even in this narrow subarea of research. We thus hasten to add references to surveys about sublineartime algorithms [16], streaming algorithms [42], and convex optimization [12]. Even for the methods mentioned here, space limitations did not allow us to go into too much detail, so we focussed on some easily accessible examples.…”

Accelerated anti-lopsided algorithm for nonnegative least squares

“…The 6 datasets, the generated dataset code and our source codes are published for in the link. 4 Environment settings: we develop Algorithm 1 in MAT-LAB with embedded code C++ to compare them with other algorithms. For NNLS, we set system parameters to use only 1 CPU for MATLAB and the IO time is excluded in the machine 8-Core Intel Xeon E5 3 GHz.…”

“…In addition, exact algorithms often lack flexibility for low-rank regularized variants and also have high complexity and slow convergence. Hence, fast approximate algorithms based on the first-order methods are more preferred to naturally provide a flexible framework for low-rank models [4,[9][10][11].…”

A Multi-commodity Network Flow Model for Cloud Service Environments

“…Efficient numerical methods to obtain x in the context of large scale problems arising in big data applications are, namely, first order methods, randomization as well as parallel and distributed computing [8].…”