On the exact minimization of saturated loss functions for robust regression and subspace estimation

Lauer, Fabien

doi:10.1016/j.patrec.2018.08.004

Cited by 4 publications

(2 citation statements)

References 16 publications

(25 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The related studies and applications of truncated least squares regression, to name a few, can be found in Hinich and Talwar (1975), Yang and Ness (1995), Ikami, Yamasaki, and Aizawa (2018), Lauer (2018), and Liu and Jiang (2019).…”

Section: Motivating Scenariosmentioning

confidence: 99%

A Framework of Learning Through Empirical Gain Maximization

Feng

2021

Neural Computation

View full text Add to dashboard Cite

We develop in this letter a framework of empirical gain maximization (EGM) to address the robust regression problem where heavy-tailed noise or outliers may be present in the response variable. The idea of EGM is to approximate the density function of the noise distribution instead of approximating the truth function directly as usual. Unlike the classical maximum likelihood estimation that encourages equal importance of all observations and could be problematic in the presence of abnormal observations, EGM schemes can be interpreted from a minimum distance estimation viewpoint and allow the ignorance of those observations. Furthermore, we show that several well-known robust nonconvex regression paradigms, such as Tukey regression and truncated least square regression, can be reformulated into this new framework. We then develop a learning theory for EGM by means of which a unified analysis can be conducted for these well-established but not fully understood regression approaches. This new framework leads to a novel interpretation of existing bounded nonconvex loss functions. Within this new framework, the two seemingly irrelevant terminologies, the well-known Tukey's biweight loss for robust regression and the triweight kernel for nonparametric smoothing, are closely related. More precisely, we show that Tukey's biweight loss can be derived from the triweight kernel. Other frequently employed bounded nonconvex loss functions in machine learning, such as the truncated square loss, the Geman-McClure loss, and the exponential squared loss, can also be reformulated from certain smoothing kernels in statistics. In addition, the new framework enables us to devise new bounded nonconvex loss functions for robust learning.

show abstract

Section: Motivating Scenariosmentioning

confidence: 99%

A Framework of Learning Through Empirical Gain Maximization

Feng

2021

Neural Computation

View full text Add to dashboard Cite

show abstract

“…The related studies and applications of truncated least squares regression, to name a few, can be found in [29,59,33,36,39].…”

Section: Motivating Scenariosmentioning

confidence: 99%

A Framework of Learning Through Empirical Gain Maximization

Feng

2020

Preprint

View full text Add to dashboard Cite

We develop in this paper a framework of empirical gain maximization (EGM) to address the robust regression problem where heavy-tailed noise or outliers may present in the response variable. The idea of EGM is to approximate the density function of the noise distribution instead of approximating the truth function directly as usual. Unlike the classical maximum likelihood estimation that encourages equal importance of all observations and could be problematic in the presence of abnormal observations, EGM schemes can be interpreted from a minimum distance estimation viewpoint and allow the ignorance of those observations. Furthermore, it is shown that several well-known robust nonconvex regression paradigms, such as Tukey regression and truncated least square regression, can be reformulated into this new framework. We then develop a learning theory for EGM, by means of which a unified analysis can be conducted for these well-established but not fully-understood regression approaches. Resulting from the new framework, a novel interpretation of existing bounded nonconvex loss functions can be concluded. Within this new framework, the two seemingly irrelevant terminologies, the wellknown Tukey's biweight loss for robust regression and the triweight kernel for nonparametric smoothing, are closely related. More precisely, it is shown that the Tukey's biweight loss can be derived from the triweight kernel. Similarly, other frequently employed bounded nonconvex loss functions in machine learning such as the truncated square loss, the Geman-McClure loss, and the exponential squared loss can also be reformulated from certain smoothing kernels in statistics. In addition, the new framework enables us to devise new bounded nonconvex loss functions for robust learning.

show abstract

Outlook

Lauer¹,

Bloch²

2018

Hybrid System Identification

View full text Add to dashboard Cite

On the exact minimization of saturated loss functions for robust regression and subspace estimation

Cited by 4 publications

References 16 publications

A Framework of Learning Through Empirical Gain Maximization

A Framework of Learning Through Empirical Gain Maximization

A Framework of Learning Through Empirical Gain Maximization

Outlook

Contact Info

Product

Resources

About