Task-Customized Self-Supervised Pre-training with Scalable Dynamic Routing

Abstract

Self-supervised learning (SSL), especially contrastive methods, has raised attraction recently as it learns effective transferable representations without semantic annotations. A common practice for self-supervised pre-training is to use as much data as possible. For a specific downstream task, however, involving irrelevant data in pre-training may degenerate the downstream performance, observed from our extensive experiments. On the other hand, for existing SSL methods, it is burdensome and infeasible to use different downstream-task-customized datasets in pre-training for different tasks. To address this issue, we propose a novel SSL paradigm called Scalable Dynamic Routing (SDR), which can be trained once and deployed efficiently to different downstream tasks with task-customized pre-trained models. Specifically, we construct the SDRnet with various sub-nets and train each sub-net with only one subset of the data by data-aware progressive training. When a downstream task arrives, we route among all the pre-trained sub-nets to get the best along with its corresponding weights. Experiment results show that our SDR can train 256 sub-nets on ImageNet simultaneously, which provides better transfer performance than a unified model trained on the full ImageNet, achieving state-of-the-art (SOTA) averaged accuracy over 11 downstream classification tasks and AP on PASCAL VOC detection task.

Strategies for effectively modelling promoter-driven gene expression using transfer learning

“…lower performance of fine-tuned models compared to models trained from scratch on the new experimental data. Negative transfer has also been observed in other fields (Liu et al, 2022), and indicates that care must be taken when choosing pretraining tasks.…”

“…Ji et al (2021); Mo et al (2021); Benegas et al (2022); Zeng et al (2023)). Pretraining using task-relevant data can improve the performance of fine-tuned models (Gururangan et al, 2020), while pretraining using irrelevant data can hurt performance (Liu et al, 2022). For our application, there are many datasets that are closely related to promoter-driven expression, including MPRAs and endogenous gene expression datasets, as well as TF-binding data that may help models learn relevant sequence motifs that regulate expression when present in promoters.…”

“…Ji et al (2021); Mo et al (2021); Benegas et al (2022); Zeng et al (2023)). Pretraining using task-relevant data can improve the performance of fine-tuned models (Gururangan et al, 2020), while pretraining using irrelevant data can hurt performance (Liu et al, 2022). For our application, many datasets are closely related to PE and are potentially useful for transfer learning.…”

Decomposed adversarial domain generalization

“…For example, generative-based SSL such as the Masked autoencoder (He et al, 2021) learns to reconstruct images with only a small fraction of the pixels. Additionally, contrastive learning methods have been observed to suffer from the negative transfer phenomenon (Liu et al, 2022), where the learned features perform poorly on downstream tasks. Furthermore, the use of cosine similarity in contrastive learning has been noted to result in overly complex feature maps (Hu et al, 2022a), which can negatively impact out-of-distribution generalization.…”

Self-supervised learning for early detection of neurodegenerative diseases with small data

“…For data that is already split at subject level, we can apply self-supervised learning directly [459]. When the data is not split, we can apply clustering to find subgroups in the data to apply self-supervised learning on [460,461]. Zhou et al [462] proposed a personalized search model which learns personalized preference based on user-level contrastive learning (e.g., users that click on the same document form a positive pair).…”