Stable Prediction with Model Misspecification and Agnostic Distribution Shift

Abstract

For many machine learning algorithms, two main assumptions are required to guarantee performance. One is that the test data are drawn from the same distribution as the training data, and the other is that the model is correctly specified. In real applications, however, we often have little prior knowledge on the test data and on the underlying true model. Under model misspecification, agnostic distribution shift between training and test data leads to inaccuracy of parameter estimation and instability of prediction across unknown test data. To address these problems, we propose a novel Decorrelated Weighting Regression (DWR) algorithm which jointly optimizes a variable decorrelation regularizer and a weighted regression model. The variable decorrelation regularizer estimates a weight for each sample such that variables are decorrelated on the weighted training data. Then, these weights are used in the weighted regression to improve the accuracy of estimation on the effect of each variable, thus help to improve the stability of prediction across unknown test data. Extensive experiments clearly demonstrate that our DWR algorithm can significantly improve the accuracy of parameter estimation and stability of prediction with model misspecification and agnostic distribution shift.

Debiased Graph Neural Networks with Agnostic Label Selection Bias

“…First, because biased labeled nodes have biased neighborhood structure and features, GNNs will encode this biased information into the node embeddings, which is validated by the experimental investigation. Based on stable learning technique [5], we make the following assumption: Assumption 1. The node embeddings learned by GNNs for each node can be decomposed as H = {S, V}, where S represents the stable variables and V represents the unstable variables.…”

“…According to the derivation rule of partitioned regression model [5], [11], with S = S and V = V, we have:…”

“…To decorrelate V and S (or g(S)), we should decorrelate the output variables of Ĝ (X, A; θ g ). [5] proposes a Variable Decorrelation (VD) term with sample reweighting technique to eliminate the correlation between each variable pair, in which the sample weights are learned by jointly minimizing the moment discrepancy between each variable pair:…”

“…Further, we theoretically analyze how the data selection bias results in the estimation bias in GNN parameters (see Section II-B). Based on the stable learning technique [5], we can assume that the learned embeddings consist of two parts: stable variables and unstable variables. The data selection bias will cause spurious correlation between these two kinds of variables.…”

“…Based on the causal view analysis of the decorrelation regularizer, we theoretically prove that the weights of variables can be differentiated by the regression coefficients. Compared with existing decorrelation methods [5], [6], the proposed regularizer is able to remove the spurious correlation while maintaining a higher effective sample size and requiring less prior knowledge. Moreover, to better combine the decorrelation regularizer with existing GNN architecture, the theoretical result shows that adding the regularizer to the embeddings learned by the penultimate layer could be both theoretically sound and flexible.…”