Suboptimality of constrained least squares and improvements via non-linear predictors

Exponential Savings in Agnostic Active Learning Through Abstention

“…We focus on the sharp analysis of ERM in a convex reference set W. Nevertheless, recent work has demonstrated that the use of so-called improper estimators -estimators that predict values outside of the reference set W -can lead to significant improvements in the dependence on some parameters. These estimators are typically based on aggregating the functions within certain non-convex reference sets constructed by truncating the original functions that correspond to W. Such improvements are known in both logistic regression (Kakade and Ng, 2004;Foster et al, 2018;Mourtada and Gaïffas, 2022) and regression with the squared loss (Forster and Warmuth, 2002;Vaškevičius and Zhivotovskiy, 2023;Mourtada et al, 2021). However, it should be noted that the improper learning setup poses a considerable challenge for the localized analysis utilized in this paper.…”