Efficient improper learning for online logistic regression

Abstract

We consider the setting of online logistic regression and consider the regret with respect to the 2 -ball of radius B. It is known (see [Hazan et al., 2014]) that any proper algorithm which has logarithmic regret in the number of samples (denoted n) necessarily suffers an exponential multiplicative constant in B. In this work, we design an efficient improper algorithm that avoids this exponential constant while preserving a logarithmic regret. Indeed, [Foster et al., 2018] showed that the lower bound does not apply to improper algorithms and proposed a strategy based on exponential weights with prohibitive computational complexity. Our new algorithm based on regularized empirical risk minimization with surrogate losses satisfies a regret scaling as O(B log(Bn)) with a per-round time-complexity of order O(d 2 ).

On Misspecification in Prediction Problems and Robustness via Improper Learning

“…Yet these algorithms are still computationally intensive; assuming our theoretical results are predictive of actual performance, one might expect that aggregating-type strategies could still yield improvements over standard empirical risk minimization. Indeed, Jézéquel et al [27] take the computational difficulty of the approximating algorithms in the paper [14] as motivation to develop an efficient improper learning algorithm for the special case of logistic regression, which (roughly) hedges its predictions by pretending to receive both positive and negative examples in future time steps, constructing a loss that depends explicitly on the new data x t ; Jézéquel et al show that it achieves a regret bound with a multiplicative RB factor of the logarithmic regret in Corollary 4. It is unclear how to extend this approach to situations in which the cardinality |Y| of Y is much larger than 1, though this is an interesting question for future work.…”

“…Thus, we seek η such that 1 2p 3/2 ≥ η(1 − 1/ η(p 3/2 − 2p + √ p)for all p ∈ [0, 1]. Letting β = √ p and solving for the stationary points ofβ 3 − 2β + β at √ p = β = 1/we see it is sufficient that 1 ≥ 2η(1/27 − 2/9 + 1/3) = 8 27 η, or η ≤27 For L quad , we have h(p) = 1 , so it suffices that I − η(p − e y )(p − e y ) T 0, or η ≤ 1 2 .…”

Non-stationary Online Regression

“…Below, we provide two results from Jézéquel et al [16] for online kernel regression with square loss. Kernel-AWV Jézéquel et al [17] computes the following estimator.…”

“…Theorem 8 (Proposition 1,[16]) Let λ, Y ≥ 0, X ⊂ R d and Y ⊂ [−Y, Y ].For any RKHS H, for n ≥ 1, for any arbitrary sequence of observations (x 1 , y 1 ), • • • , (x n , y n ) ∈ X × Y, the regret of Kernel-AWV (Equation (18),[17]) is upper-bounded for all θ ∈ H as…”

“…Theorem 10 (Restatement of Theorem 3) For online kernel regression with square loss if for all t ∈ [n], y t ∈ [−Y, Y ], then for the Alg. 2 with Kernel-AWV[17] as subroutine with parameter λ > 0, we have for all 1 ≤ r ≤ s ≤ n and all θ ∈ H…”