Extending the WILDS Benchmark for Unsupervised Adaptation

Sagawa, Shiori; Koh, Pang Wei; Lee, Tong; Gao, Irena; Xie, Sang Michael; Shen, Kendrick; Kumar, Ananya; Hu, Weihua; Yasunaga, Michihiro; Marklund, Henrik; Beery, Sara; David, Etienne; Stavness, Ian; Guo, Wei; Leskovec, Jure; Saenko, Kate; Hashimoto, Tatsunori; Levine, Sergey; Finn, Chelsea; Liang, Percy

doi:10.48550/arxiv.2112.05090

Cited by 4 publications

(8 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A total of 10 datasets are included, covering shifts across cameras for wildlife monitoring, hospitals for tumor identification, users for product rating estimation, andƒ scaffolds for biochemical property prediction, etc. Sagawa et al (2021) further extend this database to include unlabeled data for unsupervised domain adaptation.…”

Section: General Ood Databasesmentioning

confidence: 99%

DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery -- A Focus on Affinity Prediction Problems with Noise Annotations

Ji¹,

Zhang²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

AI-aided drug discovery (AIDD) is gaining increasing popularity due to its promise of making the search for new pharmaceuticals quicker, cheaper and more efficient. In spite of its extensive use in many fields, such as ADMET prediction, virtual screening, protein folding and generative chemistry, little has been explored in terms of the out-ofdistribution (OOD) learning problem with noise, which is inevitable in real world AIDD applications.In this work, we present DrugOOD 1 , a systematic OOD dataset curator and benchmark for AI-aided drug discovery, which comes with an open-source Python package that fully automates the data curation and OOD benchmarking processes. We focus on one of the most crucial problems in AIDD: drug target binding affinity prediction, which involves both macromolecule (protein target) and small-molecule (drug compound). In contrast to only providing fixed datasets, DrugOOD offers automated dataset curator with user-friendly customization scripts, rich domain annotations aligned with biochemistry knowledge, realistic noise annotations and rigorous benchmarking of state-of-the-art OOD algorithms. Since the molecular data is often modeled as irregular graphs using graph neural network (GNN) backbones, DrugOOD also serves as a valuable testbed for graph OOD learning problems. Extensive empirical studies have shown a significant performance gap between in-distribution and out-of-distribution experiments, which highlights the need to develop better schemes that can allow for OOD generalization under noise for AIDD.

show abstract

Section: General Ood Databasesmentioning

confidence: 99%

DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery -- A Focus on Affinity Prediction Problems with Noise Annotations

Ji¹,

Zhang²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…For retinal OCT image segmentation, we include three UDA methods that have been employed to adapt OCT images in [21], that are the image translation methods UDA-ST [45] and CycleGAN [46], and the widely used output space adaptation method UDAS [47]. For histopathological image classification, we compare with UDA-SwAV [41], which is the best reported UDA model for histopathological image classification using the datasets as ours. For prostate segmentation, we also employ UDAS for comparison.…”

Section: B Comparison With State-of-the-art Methodsmentioning

confidence: 99%

“…For prostate segmentation, we also employ UDAS for comparison. The results of ATTA, UDA-ST on OCT images and the results of UDA-SwAV on unseen histopathological dataset A are directly referenced from [21] and [41] respectively since the same datasets, network backbones, and data split are used in their methods and ours. The other results are obtained by re-implementing based on the released code with the network backbone being consistent for all the comparison methods.…”

Section: B Comparison With State-of-the-art Methodsmentioning

confidence: 99%

“…2) Implementation Details: The choice of network architecture in our method is flexible. Without loss of generality, we followed [21] to employ a U-Net [40] for retinal OCT image segmentation, followed [41] to use a DenseNet-121 [42] pretrained on ImageNet for histopathological image classification, and employed a 3D U-Net [43] for prostate MRI image segmentation. The encoders of our 2D U-Net and 3D U-Net contain 4 Convolution-BatchNorm-ReLU blocks, which continuously down-sample the image resolution and double the feature channels dimensions.…”

Section: B Dynamic Learning Rate On Test Datamentioning

confidence: 99%

See 1 more Smart Citation

DLTTA: Dynamic Learning Rate for Test-time Adaptation on Cross-domain Medical Images

Yang¹,

Chen²,

Jiang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Test-time adaptation (TTA) has increasingly been an important topic to efficiently tackle the cross-domain distribution shift at test time for medical images from different institutions. Previous TTA methods have a common limitation of using a fixed learning rate for all the test samples. Such a practice would be sub-optimal for TTA, because test data may arrive sequentially therefore the scale of distribution shift would change frequently. To address this problem, we propose a novel dynamic learning rate adjustment method for test-time adaptation, called DLTTA, which dynamically modulates the amount of weights update for each test image to account for the differences in their distribution shift. Specifically, our DLTTA is equipped with a memory bank based estimation scheme to effectively measure the discrepancy of a given test sample. Based on this estimated discrepancy, a dynamic learning rate adjustment strategy is then developed to achieve a suitable degree of adaptation for each test sample. The effectiveness and general applicability of our DLTTA is extensively demonstrated on three tasks including retinal optical coherence tomography (OCT) segmentation, histopathological image classification, and prostate 3D MRI segmentation. Our method achieves effective and fast test-time adaptation with consistent performance improvement over current stateof-the-art test-time adaptation methods. Code is available at: https://github.com/med-air/DLTTA.

show abstract

“…Koh et al [51] introduced WILDS: a benchmark of several new in-the-wild distribution shifts datasets across diverse data modalities, i.e., IWildCam2020-WILDS, Camelyon17-WILDS, RxRx1-WILDS, OGB-MolPCBA, GlobalWheat-WILDS, CivilComments-WILDS, FMoW-WILDS, PovertyMap-WILDS, Amazon-WILDS, and Py150-WILDS. WILDS was recently extended with unlabeled samples for multiple of its datasets [90].…”

Section: Existing Benchmarks For Ood Generalizationmentioning

confidence: 99%

WOODS: Benchmarks for Out-of-Distribution Generalization in Time Series Tasks

Gagnon-Audet¹,

Ahuja²,

Darvishi-Bayazi³

et al. 2022

Preprint

View full text Add to dashboard Cite

Machine learning models often fail to generalize well under distributional shifts. Understanding and overcoming these failures have led to a research field of Out-of-Distribution (OOD) generalization. Despite being extensively studied for static computer vision tasks, OOD generalization has been underexplored for time series tasks. To shine light on this gap, we present WOODS: eight challenging open-source time series benchmarks covering a diverse range of data modalities, such as videos, brain recordings, and sensor signals. We revise the existing OOD generalization algorithms for time series tasks and evaluate them using our systematic framework. Our experiments show a large room for improvement for empirical risk minimization and OOD generalization algorithms on our datasets, thus underscoring the new challenges posed by time series tasks. Code and documentation are available at https://woods-benchmarks.github.io/

show abstract

Extending the WILDS Benchmark for Unsupervised Adaptation

Cited by 4 publications

References 60 publications

DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery -- A Focus on Affinity Prediction Problems with Noise Annotations

DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery -- A Focus on Affinity Prediction Problems with Noise Annotations

DLTTA: Dynamic Learning Rate for Test-time Adaptation on Cross-domain Medical Images

WOODS: Benchmarks for Out-of-Distribution Generalization in Time Series Tasks

Contact Info

Product

Resources

About