Finite-sample analysis of $M$-estimators using self-concordance

Abstract

The classical asymptotic theory for parametric M -estimators guarantees that, in the limit of infinite sample size, the excess risk has a chi-square type distribution, even in the misspecified case. We demonstrate how self-concordance of the loss allows to characterize the critical sample size sufficient to guarantee a chi-square type in-probability bound for the excess risk. Specifically, we consider two classes of losses: (i) self-concordant losses in the classical sense of Nesterov and Nemirovski, i.e., whose third derivative is uniformly bounded with the 3/2 power of the second derivative; (ii) pseudo self-concordant losses, for which the power is removed. These classes contain losses corresponding to several generalized linear models, including the logistic loss and pseudo-Huber losses.Our basic result under minimal assumptions bounds the critical sample size by O(d • d eff ), where d the parameter dimension and d eff the effective dimension that accounts for model misspecification. In contrast to the existing results, we only impose local assumptions that concern the population risk minimizer θ * . Namely, we assume that the calibrated predictors, i.e., predictors scaled by the square root of the second derivative of the loss, is subgaussian at θ * . Besides, for type-ii losses we require boundedness of certain measure of curvature of the population risk at θ * .Our improved result bounds the critical sample size from above as O(max{d eff , d log d})under slightly stronger assumptions. Namely, the local assumptions must hold in the neighborhood of θ * given by the Dikin ellipsoid of the population risk. Interestingly, we find that, for logistic regression with Gaussian design, there is no actual restriction of conditions: the subgaussian parameter and curvature measure remain near-constant over the Dikin ellipsoid. Finally, we extend some of these results to 1 -penalized estimators in high dimensions.

Simple steps are all you need: Frank-Wolfe and generalized self-concordant functions

“…For example, the logistic loss function used in logistic regression is not strictly self-concordant, but it fits into a class of pseudo-self-concordant functions, which allows one to obtain similar properties and bounds as those obtained for selfconcordant functions [Bach et al, 2010]. This was also the case in Ostrovskii & Bach [2021] and Tran-Dinh et al [2015], in which more general properties of these pseudo-self-concordant functions were established. This was fully formalized in Sun & Tran-Dinh [2019], in which the concept of generalized-self concordant functions was introduced, along with key bounds, properties, and variants of Newton methods for the unconstrained setting which make use of this property.…”

“…Self-concordant functions have received strong interest in recent years due to the attractive properties that they allow to prove for many statistical estimation settings [Marteau-Ferey et al, 2019, Ostrovskii & Bach, 2021. The original definition of self-concordance has been expanded and generalized since its inception, as many objective functions of interest have self-concordant-like properties without satisfying the strict definition of self-concordance.…”

Convergence Rates for the MAP of an Exponential Family and Stochastic Mirror Descent -- an Open Problem

“…Therefore, the expected KL ( 13) is infinite for any number of samples. Instead of taking the expectation, one might want to bound the risk in high probability without resorting to Markov inequality, as achieved by (Ostrovskii and Bach, 2021), but this is a difficult endeavor. These examples make a case for regularized estimators such as MAP, for which we may find upper bounds.…”

“…Overcoming this limitation, Ostrovskii and Bach (2021) characterize the number of samples needed to be upper bounded by a quadratic with high-probability, for any parametric models with a self-concordant log-likelihood f . Anastasiou and Reinert (2017) obtains a similar flavor of result under other assumptions on the third derivative of f .…”

“…Despite the MLE and MAP estimators in the exponential family being classical and known in statistics for decades, we highlighted in this paper open problems to bound their frequentist risk (the expected KL) in a non-asymptotic way. We reviewed some partial results, such as a large sample analysis that describes how many samples are needed to ensure a locally quadratic regime (Kakade et al, 2010;Ostrovskii and Bach, 2021) for which rates are known. We also related this problem to the one of obtaining convergence rates in stochastic optimization, observing that MAP fits the framework of stochastic mirror descent with relative smoothness assumptions.…”