Linear Regression With Shuffled Data: Statistical and Computational Limits of Permutation Recovery

Pananjady, Ashwin; Wainwright, Martin J.; Courtade, Thomas A.

doi:10.1109/tit.2017.2776217

Cited by 89 publications

(97 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Estimation of the permutation matrix Π is challenging both computationally and statistically. Specifically, permutation recovery is generally NP-hard unless p = 1 or U i = 0 (Pananjady et al 2017b, Hsu et al 2017). When p = 1, estimation of Π reduces to a sorting problem and thus is computationally tractable.…”

Section: Spherical Regression With Mismatched Datamentioning

confidence: 99%

Spherical Regression Under Mismatch Corruption With Application to Automated Knowledge Translation

Shi

Cai

2020

Journal of the American Statistical Association

View full text Add to dashboard Cite

Motivated by a series of applications in data integration, language translation, bioinformatics, and computer vision, we consider spherical regression with two sets of unit-length vectors when the data are corrupted by a small fraction of mismatch in the response-predictor pairs. We propose a three-step algorithm in which we initialize the parameters by solving an orthogonal Procrustes problem to estimate a translation matrix W ignoring the mismatch. We then estimate a mapping matrix aiming to correct the mismatch using hard-thresholding to induce sparsity, while incorporating potential group information. We eventually obtain a refined estimate for W by removing the estimated mismatched pairs. We derive the error bound for the initial estimate of W in both fixed and high-dimensional setting. We demonstrate that the refined estimate of W achieves an error rate that is as good as if no mismatch is present. We show that our mapping recovery method not only correctly distinguishes one-to-one and one-to-many correspondences, but also consistently identifies the matched pairs and estimates the weight vector for combined correspondence. We examine the finite sample performance of the proposed method via extensive simulation studies, and with application to the unsupervised translation of medical codes using electronic health records data.

show abstract

Section: Spherical Regression With Mismatched Datamentioning

confidence: 99%

Spherical Regression Under Mismatch Corruption With Application to Automated Knowledge Translation

Shi

Cai

2020

Journal of the American Statistical Association

View full text Add to dashboard Cite

show abstract

“…Over the past years there has been a considerable amount of work on instances of Problem 1 that come with additional structure in diverse contexts, e.g., see the excellent literature reviews in [9], [10], [18]. Nevertheless, it has only been until very recently that the problem of shuffled linear regression has been considered in its full generality.…”

Section: Prior Artmentioning

confidence: 99%

“…Nevertheless, it has only been until very recently that the problem of shuffled linear regression has been considered in its full generality. In fact, the main achievements so far have been concentrating on a theoretical understanding of the conditions that allow unique recovery of ξ * or Π * ; see [8], [28], [9], [10], [29], [11], [31], [32], [33], [18], [34], [35], [36], [19]. Letting A be drawn at random from any continuous probability distribution, [10] proved that any such ξ * can be uniquely recovered with probability 1 as long as 2 m ≥ 2n.…”

Section: Prior Artmentioning

confidence: 99%

“…intended to shed light on the computational difficulty of the least squares problem (5). Indeed, as per [9] for 3 n > 1 this is an NP-hard problem 4 . On the other hand, the approach that seems to be the predominant one in terms of practical deployment is that of solving (5) via alternating minimization [29]: given an estimate for ξ * one computes an estimate for Π * via sorting; given an estimate for Π * one computes an estimate for ξ * via ordinary least-squares.…”

Section: Prior Artmentioning

confidence: 99%

“…Finally, we note that in the setting of Theorem U-H-V unique recovery of the permutation Π * is in principle not possible, as per Theorem 10 in [10]. Instead, one has to either allow for the observed vector y to be generic (i.e., the permutation of a generic vector of R(A)) in which case Π * is uniquely recoverable by Theorem 1, or consider unique recovery with high probability, which is indeed possible even when y is corrupted by noise [9].…”

Section: The Notion Of Generic a Ymentioning

confidence: 99%

See 2 more Smart Citations

An Algebraic-Geometric Approach for Linear Regression Without Correspondences

Tsakiris

Peng

Conca

et al. 2020

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

Linear regression without correspondences is the problem of performing a linear regression fit to a dataset for which the correspondences between the independent samples and the observations are unknown. Such a problem naturally arises in diverse domains such as computer vision, data mining, communications and biology. In its simplest form, it is tantamount to solving a linear system of equations, for which the entries of the right hand side vector have been permuted. This type of data corruption renders the linear regression task considerably harder, even in the absence of other corruptions, such as noise, outliers or missing entries. Existing methods are either applicable only to noiseless data or they are very sensitive to initialization or they work only for partially shuffled data. In this paper we address these issues via an algebraic geometric approach, which uses symmetric polynomials to extract permutation-invariant constraints that the parameters ξ * ∈ R n of the linear regression model must satisfy. This naturally leads to a polynomial system of n equations in n unknowns, which contains ξ * in its root locus. Using the machinery of algebraic geometry we prove that as long as the independent samples are generic, this polynomial system is always consistent with at most n! complex roots, regardless of any type of corruption inflicted on the observations. The algorithmic implication of this fact is that one can always solve this polynomial system and use its most suitable root as initialization to the Expectation Maximization algorithm. To the best of our knowledge, the resulting method is the first working solution for small values of n able to handle thousands of fully shuffled noisy observations in milliseconds.

show abstract

Regression with linked datasets subject to linkage error

Wang

Ben-David

Diao

et al. 2021

WIREs Computational Stats

View full text Add to dashboard Cite

Data are often collected from multiple heterogeneous sources and are combined subsequently. In combing data, record linkage is an essential task for linking records in datasets that refer to the same entity. Record linkage is generally not error-free; there is a possibility that records belonging to different entities are linked or that records belonging to the same entity are missed. It is not advisable to simply ignore such errors because they can lead to data contamination and introduce bias in sample selection or estimation, which, in return, can lead to misleading statistical results and conclusions. For a long while, this problem was not properly recognized, but in recent years a growing number of researchers have developed methodology for dealing with linkage errors in regression analysis with linked datasets. The main goal of this overview is to give an account of those developments, with an emphasis on recent approaches and their connection to the so-called "Broken Sample" problem.We also provide a short empirical study that illustrates the efficacy of corrective methods in different scenarios. This article is categorized under: Statistical Models > Model Selection Statistical and Graphical Methods of Data Analysis > Robust Methods Statistical and Graphical Methods of Data Analysis > Multivariate Analysis

show abstract

Linear Regression With Shuffled Data: Statistical and Computational Limits of Permutation Recovery

Cited by 89 publications

References 29 publications

Spherical Regression Under Mismatch Corruption With Application to Automated Knowledge Translation

Spherical Regression Under Mismatch Corruption With Application to Automated Knowledge Translation

An Algebraic-Geometric Approach for Linear Regression Without Correspondences

Regression with linked datasets subject to linkage error

Contact Info

Product

Resources

About