A mathematical programming approach for improving the robustness of least sum of absolute deviations regression

Abstract

This paper discusses a novel application of mathematical programming techniques to a regression problem. While least squares regression techniques have been used for a long time, it is known that their robustness properties are not desirable. Specifically, the estimators are known to be too sensitive to data contamination. In this paper we examine regressions based on Least-sum of Absolute Deviations (LAD) and show that the robustness of the estimator can be improved significantly through a judicious choice of weights. The problem of finding optimum weights is formulated as a nonlinear mixed integer program, which is too difficult to solve exactly in general. We demonstrate that our problem is equivalent to a mathematical program with a single functional constraint resembling the knapsack problem and then solve it for a special case. We then generalize this solution to general regression designs. Furthermore, we provide an efficient algorithm to solve the general nonlinear, mixed integer programming problem when the number of predictors is small. We show the efficacy of the weighted LAD estimator using numerical examples.

Sharp non-asymptotic performance bounds for $$\ell _1$$ ℓ 1 and Huber robust regression estimators

“…Thus, the 1 estimator mitigates the effect of the largest outliers. They can also be generalized to weighted 1 regression (Giloni et al 2006a, b) by adding weights to the absolute values in the definition of the leverage constants. The use of the leverage constants to improve the RBP of weighted 1 regression is the subject of current research.…”

Robust weighted LAD regression

“…Mizera and Müller (2001) and Giloni and Padberg (2004) showed that the finite sample breakdown point of LAD regression can be greater than 1/n, depending on the predictor values, with the former authors discussing how X can be chosen to increase the breakdown point of LAD. Giloni and Padberg (2004) showed that the breakdown can be calculated using a mixed integer program, and described an algorithm for solving it; Giloni, Sengupta, and Simonoff (2004) proposed an alternative algorithm similar to that of Mizera and Müller (2001) that is very efficient for large samples when p is small. Ellis and Morgenthaler (1992) appear to be the first to mention that the introduction of weights can improve the finite sample breakdown point of LAD regression (by downweighting observations that are far from the bulk of the data), but they only show this for very small data sets.…”

“…Ellis and Morgenthaler (1992) appear to be the first to mention that the introduction of weights can improve the finite sample breakdown point of LAD regression (by downweighting observations that are far from the bulk of the data), but they only show this for very small data sets. Giloni, Sengupta, and Simonoff (2004) examined this question in more detail. The weighted LAD (WLAD) regression estimator is defined to be the minimizer of…”

“…Giloni, Sengupta, and Simonoff (2004) showed that this problem is equivalent to a problem related to the knapsack problem, and solved a specific form of the problem for the simple regression case with a uniform design, resulting in a WLAD breakdown over 30% (LAD regression has a 25% breakdown point in this situation). They also proposed a simple weighting scheme for multiple regression that yields breakdown points over 20% for various two-predictor problems.…”

Prospective Analysis of Distribution Network Reconstruction on Electric Vehicles access to Demonstration District

“…Reference [4] mainly researched on the errors between LAD and Least Square method. Reference [5] improved the robustness of LAD regression through a judicious choice of weights. Furthermore, it also provided an efficient algorithm to solve the general nonlinear, mixed integer programming problem when the number of predictors is small.…”