Causal Machine Learning: A Survey and Open Problems

Jean, Kaddour,; Lynch, Aengus; Liu, Qi; Kusner, Matt J.; Silva, Ricardo

doi:10.48550/arxiv.2206.15475

Cited by 21 publications

(28 citation statements)

References 171 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The inner mechanism of the labeling rule G → Y usually depends on the causal features, which are particular subparts of the entire data (Arjovsky et al, 2019;Kaddour et al, 2022;Wu et al, 2022b;Ye et al, 2022;Lu et al, 2021). While their complementary parts, environmental features, are noncausal for predicting the graphs.…”

Section: Definitions and Problem Formationsmentioning

confidence: 99%

“…Hence, the distribution of augmented data tends not to overlap with the original distribution, which encourages the diversity of environmental features. However, from a data generation perspective, causal features are invariant and shared across environments (Kaddour et al, 2022), so they are essential features for OOD generalization. Since Principle 1 does not expose any constraint on the invariant property of the augmented distribution, we here propose the second principle for augmentation:…”

Section: Two Principles For Graph Augmentationmentioning

confidence: 99%

See 1 more Smart Citation

Adversarial Causal Augmentation for Graph Covariate Shift

Sui¹,

Wang²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Out-of-distribution (OOD) generalization on graphs is drawing widespread attention. However, existing efforts mainly focus on the OOD issue of correlation shift. While another type, covariate shift, remains largely unexplored but is the focus of this work. From a data generation view, causal features are stable substructures in data, which play key roles in OOD generalization. While their complementary parts, environments, are unstable features that often lead to various distribution shifts. Correlation shift establishes spurious statistical correlations between environments and labels. In contrast, covariate shift means that there exist unseen environmental features in test data. Existing strategies of graph invariant learning and data augmentation suffer from limited environments or unstable causal features, which greatly limits their generalization ability on covariate shift. In view of that, we propose a novel graph augmentation strategy: Adversarial Causal Augmentation (AdvCA), to alleviate the covariate shift. Specifically, it adversarially augments the data to explore diverse distributions of the environments. Meanwhile, it keeps the causal features invariant across diverse environments. It maintains the environmental diversity while ensuring the invariance of the causal features, thereby effectively alleviating the covariate shift. Extensive experimental results with in-depth analyses demonstrate that AdvCA can outperform 14 baselines on synthetic and real-world datasets with various covariate shifts.

show abstract

Section: Definitions and Problem Formationsmentioning

confidence: 99%

Section: Two Principles For Graph Augmentationmentioning

confidence: 99%

Adversarial Causal Augmentation for Graph Covariate Shift

Sui¹,

Wang²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Another line of work has shown that training data re-weighting can speed up training. For example, some re-weighting methods focus on proxy models [2,29], importance sampling [3,19] or removing spurious correlations [17,36].…”

Section: Related Workmentioning

confidence: 99%

Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging

Jean¹

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Training vision or language models on large datasets can take days, if not weeks. We show that averaging the weights of the k latest checkpoints, each collected at the end of an epoch, can speed up the training progression in terms of loss and accuracy by dozens of epochs, corresponding to time savings up to 68 and 30 GPU hours when training a ResNet50 on ImageNet and RoBERTa-Base model on WikiText-103, respectively. We also provide the code and model checkpoint trajectory to reproduce the results and facilitate research on reusing historical weights for faster convergence 1 . However, in deep learning, loss functions are highly non-convex [8]. Weight averaging has been mainly used to improve the model's generalization performance at the end of or after training [14,16].Contribution We revisit weight averaging applied to neural networks from a convergence speed perspective. Inspired by Li et al. [22], we focus on the middle stage of training: after the dramatic changes of the local loss landscape during the very first training steps [10,7] but before the optimizer converges. Because the weights still undergo substantial change in that middle phase, averaging all 1 https://github.com/jeankaddour/lawa Preprint. Under review.

show abstract

“…In many domains, including cell biology (Sachs et al, 2005), finance (Sanford & Moosa, 2012), and genetics (Zhang et al, 2013), the data generating process is thought to be represented by an underlying directed acylic graph (DAG). Many models rely on DAG assumptions, e.g., causal modeling uses DAGs to model distribution shifts, ensure predictor fairness among subpopulations, or learn agents more sample-efficiently (Kaddour et al, 2022). A key question, with implications ranging from better modeling to causal discovery, is how to recover this unknown DAG from observed data alone.…”

Section: Introductionmentioning

confidence: 99%

DAG Learning on the Permutahedron

Zantedeschi¹,

Franceschi²,

Jean³

et al. 2023

Preprint

View full text Add to dashboard Cite

We propose a continuous optimization framework for discovering a latent directed acyclic graph (DAG) from observational data. Our approach optimizes over the polytope of permutation vectors, the so-called Permutahedron, to learn a topological ordering. Edges can be optimized jointly, or learned conditional on the ordering via a non-differentiable subroutine. Compared to existing continuous optimization approaches our formulation has a number of advantages including: 1. validity: optimizes over exact DAGs as opposed to other relaxations optimizing approximate DAGs; 2. modularity: accommodates any edge-optimization procedure, edge structural parameterization, and optimization loss; 3. end-to-end: either alternately iterates between node-ordering and edge-optimization, or optimizes them jointly. We demonstrate, on real-world data problems in protein-signaling and transcriptional network discovery, that our approach lies on the Pareto frontier of two key metrics, the SID and SHD.

show abstract

Causal Machine Learning: A Survey and Open Problems

Cited by 21 publications

References 171 publications

Adversarial Causal Augmentation for Graph Covariate Shift

Adversarial Causal Augmentation for Graph Covariate Shift

Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging

DAG Learning on the Permutahedron

Contact Info

Product

Resources

About