Logarithmic regret algorithms for online convex optimization

Abstract

In an online convex optimization problem a decision-maker makes a sequence of decisions, i.e., chooses a sequence of points in Euclidean space, from a fixed feasible set. After each point is chosen, it encounters a sequence of (possibly unrelated) convex cost functions. Zinkevich (ICML 2003) introduced this framework, which models many natural repeated decision-making problems and generalizes many existing problems such as Prediction from Expert Advice and Cover's Universal Portfolios. Zinkevich showed that a simple online gradient descent algorithm achieves additive regret O( √ T ), for an arbitrary sequence of T convex cost functions (of bounded gradients), with respect to the best single decision in hindsight.In this paper, we give algorithms that achieve regret O(log(T )) for an arbitrary sequence of strictly convex functions (with bounded first and second derivatives). This mirrors what has been done for the special cases of prediction from expert advice by Kivinen and Warmuth (EuroCOLT 1999), and Universal Portfolios by Cover (Math. Finance 1: [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19] 1991). We propose several algorithms achieving logarithmic regret, which besides being more general are also much more efficient to implement.The main new ideas give rise to an efficient algorithm based on the Newton method for optimization, a new tool in the field. Our analysis shows a surprising connection between the natural follow-the-leader approach and the Newton method. We also analyze other algorithms, which tie together several different previous approaches including follow-the-leader, exponential weighting, Cover's algorithm and gradient descent.

A Nonparametric Asymptotic Analysis of Inventory Planning with Censored Demand

“…Therefore, it follows that E [ g(w t ), w t − w * ] is bounded above by the right-hand side of (15). Since the subgradient g(z) defines the supporting hyperplane of the convex function Φ at z, it follows that g(w t ), w t − w * is an upper bound on Φ(w t ) − Φ(w * ).…”

“…This result was extended by Flaxman et al [11] to the case where the optimizer instead obtains an unbiased estimator of the gradient. Under additional technical assumptions on the shape of the convex function, a modified algorithm by Hazan et al [15] achieves a faster convergence rate of O (log(T )). The case where the available information is an unbiased estimator of the objective value, not its derivative, has been studied by Flaxman et al [11] and Kleinberg [22].…”

“…In this section, we show by example that this rate can indeed be Θ(1/ √ T ). Hazan et al [15] have established such a lower bound on the convergence rate in an adversarial setting, but not for the stochastic non-adversarial setting.…”

“…Observe that most of the arguments in the proofs of Lemma 3 remain valid in this case. In particular, Equation (15) in the proof of Lemma 3 in Appendix A holds. Thus,…”

“…If we modify the step size of the original AIM algorithm so that for t = 1/ (α(b + h)t) for all t, then it follows from Theorem 1 in Hazan et al [15] that, for the perishable inventory case,…”