Boosting in the Presence of Outliers: Adaptive Classification With Nonconvex Loss Functions

Abstract

This paper examines the role and efficiency of the non-convex loss functions for binary classification problems. In particular, we investigate how to design a simple and effective boosting algorithm that is robust to the outliers in the data. The analysis of the role of a particular non-convex loss for prediction accuracy varies depending on the diminishing tail properties of the gradient of the loss -the ability of the loss to efficiently adapt to the outlying data, the local convex properties of the loss and the proportion of the contaminated data. In order to use these properties efficiently, we propose a new family of non-convex losses named γ-robust losses. Moreover, we present a new boosting framework, Arch Boost, designed for augmenting the existing work such that its corresponding classification algorithm is significantly more adaptable to the unknown data contamination. Along with the Arch Boosting framework, the non-convex losses lead to the new class of boosting algorithms, named adaptive, robust, boosting (ARB). Furthermore, we present theoretical examples that demonstrate the robustness properties of the proposed algorithms. In particular, we develop a new breakdown point analysis and a new influence function analysis that demonstrate gains in robustness. Moreover, we present new theoretical results, based only on local curvatures, which may be used to establish statistical and optimization properties of the proposed Arch boosting algorithms with highly non-convex loss functions. Extensive numerical calculations are used to illustrate these theoretical properties and reveal advantages over the existing boosting methods when data exhibits a number of outliers.

The macroecology of the COVID-19 pandemic in the Anthropocene

“…For the number of cases and deaths, we used the Poisson distribution and for the growth rate, we used the Gaussian distribution. We used natural logarithm transformation (variables: Dens, GDP, Time, Tour) because gradient boosting regression may produce biased results in the presence of outliers [ 44 ]. The fitted model was then used to make predictions on the test data.…”

All-In-One Robust Estimator of the Gaussian Mean

“…There are also many other statistical problems in which robust estimation has been recently revisited from the point of view of minimax rates. This includes scale and covariance matrix estimation (Chen et al, 2018;Comminges et al, 2018), matrix completion (Klopp et al, 2017), multivariate regression (Dalalyan and Thompson, 2019;Gao, 2020;Geoffrey, 2019), classification (Cannings et al, 2018;Li and Bradic, 2018), subspace clustering (Soltanolkotabi and Candès, 2012), community detection (Cai and Li, 2015), etc. Properties of robust M -estimators in high-dimensional settings are studied in (Elsener and van de Geer, 2018;Loh, 2017).…”

MM for Penalized Estimation

“…Shi et al ( 2010) used gene expressions to predict breast cancer clinical status with 164 estrogen receptor positive cases and 114 negative cases. The same data set has been evaluated for a variety of robust classification algorithms (Li and Bradic, 2018;Wang, 2018a,b). The analysis described below is reproduced in a vignette distributed with the R package mpath.…”

“…However, a breakdown point analysis shows that convex loss functions such as the LAD-LASSO or penalized M-estimators are not robust enough and even a single outlier can send the regression coefficients to infinity (Alfons et al, 2013). On the other hand, bounded nonconvex loss functions have shown more resistant to outliers than traditional convex loss functions in regression and classification (Liu et al, 2007;Freund, 2001;Wu and Liu, 2007;Singh et al, 2014;Chi and Scott, 2014;Li and Bradic, 2018;Wang, 2018a,b). This article is concerned with variable selection using penalized nonconvex loss functions in regression and classification.…”