Equivalence of predictors under real and over-parameterized linear models

Gan, Shengjun; Sun, Yan; Tian, Yongge

doi:10.1080/03610926.2015.1100742

Cited by 19 publications

(3 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There have been many explanations regarding this drop in accuracy Zhang et al, 2019;Fawzi et al, 2018;Nakkiran, 2019). The recent line of work on "double descent" (Bartlett et al, 2019;Gan et al, 2017;Belkin et al, 2019;Mei and Montanari, 2019) has shed some lights on this trade-off and show that unlike conventional under-parameterized regime in the new over-parameterized regime, more data and fewer parameters might result in a worse error.…”

Section: Related Work and Discussionmentioning

confidence: 99%

Removing Spurious Features can Hurt Accuracy and Affect Groups Disproportionately

Khani

Liang

2021

Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency

View full text Add to dashboard Cite

The presence of spurious features interferes with the goal of obtaining robust models that perform well across many groups within the population. A natural remedy is to remove spurious features from the model. However, in this work we show that removal of spurious features can decrease accuracy due to the inductive biases of overparameterized models. We completely characterize how the removal of spurious features affects accuracy across different groups (more generally, test distributions) in noiseless overparameterized linear regression. In addition, we show that removal of spurious feature can decrease the accuracy even in balanced dataset-each target co-occurs equally with each spurious feature; and it can inadvertently make the model more susceptible to other spurious features. Finally, we show that robust self-training can remove spurious features without affecting the overall accuracy. Experiments on the Toxic-Comment-Detectoin and CelebA datasets show that our results hold in non-linear models.

show abstract

Section: Related Work and Discussionmentioning

confidence: 99%

Removing Spurious Features can Hurt Accuracy and Affect Groups Disproportionately

Khani

Liang

2021

Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency

View full text Add to dashboard Cite

show abstract

“…In comparison, the whole results collected in Theorem 3.2 provide an exclusive and unified theory about BLUPs and BLUEs of parameter spaces and their magnificent properties and features under GLMs. As demonstrated in [22,23], the results in Theorem 3.2 can serve as basic and useful references being applied in the statistical inference of GLMs.…”

Section: Formulas and Properties Of Blupsmentioning

confidence: 99%

“…A general method for solving this kind of matrix optimization problems in the Löwner partial ordering was formulated in Lemma 2.8. In particular, the following comprehensive results were established in [22]; see also [23]. …”

Section: Formulas and Properties Of Blupsmentioning

confidence: 99%

Matrix rank and inertia formulas in the analysis of general linear models

Tian

2017

Open Mathematics

Self Cite

View full text Add to dashboard Cite

Matrix mathematics provides a powerful tool set for addressing statistical problems, in particular, the theory of matrix ranks and inertias has been developed as effective methodology of simplifying various complicated matrix expressions, and establishing equalities and inequalities occurred in statistical analysis. This paper describes how to establish exact formulas for calculating ranks and inertias of covariances of predictors and estimators of parameter spaces in general linear models (GLMs), and how to use the formulas in statistical analysis of GLMs. We first derive analytical expressions of best linear unbiased predictors/best linear unbiased estimators (BLUPs/BLUEs) of all unknown parameters in the model by solving a constrained quadratic matrix-valued function optimization problem, and present some well-known results on ordinary least-squares predictors/ordinary least-squares estimators (OLSPs/OLSEs). We then establish some fundamental rank and inertia formulas for covariance matrices related to BLUPs/BLUEs and OLSPs/OLSEs, and use the formulas to characterize a variety of equalities and inequalities for covariance matrices of BLUPs/BLUEs and OLSPs/OLSEs. As applications, we use these equalities and inequalities in the comparison of the covariance matrices of BLUPs/BLUEs and OLSPs/OLSEs. The work on the formulations of BLUPs/BLUEs and OLSPs/OLSEs, and their covariance matrices under GLMs provides direct access, as a standard example, to a very simple algebraic treatment of predictors and estimators in linear regression analysis, which leads a deep insight into the linear nature of GLMs and gives an efficient way of summarizing the results.

show abstract

Further remarks on constrained over-parameterized linear models

Güler

Büyükkaya

2023

Stat Papers

View full text Add to dashboard Cite

Equivalence of predictors under real and over-parameterized linear models

Cited by 19 publications

References 11 publications

Removing Spurious Features can Hurt Accuracy and Affect Groups Disproportionately

Removing Spurious Features can Hurt Accuracy and Affect Groups Disproportionately

Matrix rank and inertia formulas in the analysis of general linear models

Further remarks on constrained over-parameterized linear models

Contact Info

Product

Resources

About