Median-Truncated Gradient Descent: A Robust and Scalable Nonconvex Approach for Signal Estimation

Chi, Yuejie; Zhang, Huishuai; Liang, Y. T.

doi:10.1007/978-3-319-73074-5_8

Cited by 8 publications

(6 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Lipschitz constant of f i is a i ℓ 2 which motivates thinking of a Random Kaczmarz method as a basic form of stochastic gradient descent (see Needell, Srebro & Ward [38]). This analogy is also discussed in Haddock, Needell, Rebrova & Swartworth [19] who describe an analogous algorithm for SGD (see also [6,8,25,27,31]). We believe that the setting of quantile-Random Kaczmarz method applied to corrupted linear system may be a useful (because reasonably explicit) model for understanding the effect of manipulating mini-batches in SGD.…”

Section: Matrices Without Normalizationmentioning

confidence: 81%

Quantile-Based Random Kaczmarz for corrupted linear systems of equations

Steinerberger¹

2021

Preprint

View full text Add to dashboard Cite

We consider linear systems Ax = b where A ∈ R m×n consists of normalized rows, a i ℓ 2 = 1, and where up to βm entries of b have been corrupted (possibly by arbitrarily large numbers). Haddock, Needell, Rebrova & Swartworth propose a quantile-based Random Kaczmarz method and show that for certain random matrices A it converges with high likelihood to the true solution. We prove a deterministic version by constructing, for any matrix A, a number β A such that there is convergence for all perturbations with β < β A . Assuming a random matrix heuristic, this proves convergence for tall Gaussian matrices with up to ∼ 0.5% corruption (a number that can likely be improved).

show abstract

Section: Matrices Without Normalizationmentioning

confidence: 81%

Quantile-Based Random Kaczmarz for corrupted linear systems of equations

Steinerberger¹

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…An important branch of advances in the analysis of SGD deal with robustness to corruption and outliers in the objective defining data and sampled gradients, see e.g., [CLZL19,LCZL20]. Similar to our work, the aforementioned papers use quantile statistics, namely, a median-truncated SGD.…”

Section: Related Workmentioning

confidence: 82%

Quantile-based Iterative Methods for Corrupted Systems of Linear Equations

Haddock¹,

Needell²,

Rebrova³

et al. 2020

Preprint

View full text Add to dashboard Cite

Often in applications ranging from medical imaging and sensor networks to error correction and data science (and beyond), one needs to solve large-scale linear systems in which a fraction of the measurements have been corrupted. We consider solving such large-scale systems of linear equations Ax = b that are inconsistent due to corruptions in the measurement vector b. We develop several variants of iterative methods that converge to the solution of the uncorrupted system of equations, even in the presence of large corruptions. These methods make use of a quantile of the absolute values of the residual vector in determining the iterate update. We present both theoretical and empirical results that demonstrate the promise of these iterative approaches. * The authors were partially supported by NSF CAREER #1348721 and NSF BIGDATA #1740325. We also acknowledge sponsorship by Capital FundManagement.

show abstract

“…Median-truncated Loss The median-truncated loss (Chi et al, 2019) describes the loss function for subject i in subgroup k as:…”

Section: Robust Estimation Proceduresmentioning

confidence: 99%

Outcome-Guided Disease Subtyping for High-Dimensional Omics Data

Liu,

Fang,

Ren

et al. 2020

Preprint

View full text Add to dashboard Cite

High-throughput microarray and sequencing technology have been used to identify disease subtypes that could not be observed otherwise by using clinical variables alone. The classical unsupervised clustering strategy concerns primarily the identification of subpopulations that have similar patterns in gene features. However, as the features corresponding to irrelevant confounders (e.g. gender or age) may dominate the clustering process, the resulting clusters may or may not capture clinically meaningful disease subtypes. This gives rise to a fundamental problem: can we find a subtyping procedure guided by a pre-specified disease outcome? Existing methods, such as supervised clustering, apply a two-stage approach and depend on an arbitrary number of selected features associated with outcome. In this paper, we propose a unified latent generative model to perform outcome-guided disease subtyping constructed from omics data, which improves the resulting subtypes concerning the disease of interest. Feature selection is embedded in a regularization regression. A modified EM algorithm is applied for numerical computation and parameter estimation. The proposed method performs feature selection, latent subtype characterization and outcome prediction simultaneously.To account for possible outliers or violation of mixture Gaussian assumption, we incorporate robust estimation using adaptive Huber or median-truncated loss function. Extensive simulations and an application to complex lung diseases with transcriptomic and clinical data demonstrate the ability of the proposed method to identify clinically relevant disease subtypes and signature genes suitable to explore toward precision medicine.

show abstract

Median-Truncated Gradient Descent: A Robust and Scalable Nonconvex Approach for Signal Estimation

Cited by 8 publications

References 39 publications

Quantile-Based Random Kaczmarz for corrupted linear systems of equations

Quantile-Based Random Kaczmarz for corrupted linear systems of equations

Quantile-based Iterative Methods for Corrupted Systems of Linear Equations

Outcome-Guided Disease Subtyping for High-Dimensional Omics Data

Contact Info

Product

Resources

About