Robust Learning Through Cross-Task Consistency

Zamir, Amir; Sax, Alexander F.; Cheerla, Nikhil; Suri, Rohan; Cao, Zhangjie; Malik, Jitendra; Guibas, Leonidas J.

doi:10.1109/cvpr42600.2020.01121

Cited by 109 publications

(77 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, multi-task learning was recently shown to improve robustness. For example, in [115] a multi-task learning strategy showed robustness against adversarial attacks, while [116] found that applying cross-task consistency in MTL improves generalization, and allows for domain shift detection.…”

Section: Othermentioning

confidence: 99%

Multi-Task Learning for Dense Prediction Tasks: A Survey

Vandenhende¹,

Georgoulis²,

Gansbeke³

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

378

222

View full text Add to dashboard Cite

With the advent of deep learning, many dense prediction tasks, i.e. tasks that produce pixel-level predictions, have seen significant performance improvements. The typical approach is to learn these tasks in isolation, that is, a separate neural network is trained for each individual task. Yet, recent multi-task learning (MTL) techniques have shown promising results w.r.t. performance, computations and/or memory footprint, by jointly tackling multiple tasks through a learned shared representation. In this survey, we provide a well-rounded view on state-of-the-art deep learning approaches for MTL in computer vision, explicitly emphasizing on dense prediction tasks. Our contributions concern the following. First, we consider MTL from a network architecture point-of-view. We include an extensive overview and discuss the advantages/disadvantages of recent popular MTL models. Second, we examine various optimization methods to tackle the joint learning of multiple tasks. We summarize the qualitative elements of these works and explore their commonalities and differences. Finally, we provide an extensive experimental evaluation across a variety of dense prediction benchmarks to examine the pros and cons of the different methods, including both architectural and optimization based strategies.

show abstract

Section: Othermentioning

confidence: 99%

Multi-Task Learning for Dense Prediction Tasks: A Survey

Vandenhende¹,

Georgoulis²,

Gansbeke³

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

378

222

View full text Add to dashboard Cite

show abstract

“…In MTL, the goal is to reach high performance on multiple tasks simultaneously, so all tasks are main tasks and all tasks are auxiliary tasks. While the goal is different, many strategies in MTL such as parameter sharing [9], task consistency [55], and loss balance [15] are useful for learning with auxiliary tasks.…”

Section: Related Workmentioning

confidence: 99%

Hyperspectral Image Super-Resolution with RGB Image Super-Resolution as an Auxiliary Task

Dai²,

Gool

2022

2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

View full text Add to dashboard Cite

This work studies Hyperspectral image (HSI) superresolution (SR). HSI SR is characterized by highdimensional data and a limited amount of training examples. This raises challenges for training deep neural networks that are known to be data hungry. This work addresses this issue with two contributions. First, we observe that HSI SR and RGB image SR are correlated and develop a novel multi-tasking network to train them jointly so that the auxiliary task RGB image SR can provide additional supervision and regulate the network training. Second, we extend the network to a semi-supervised setting so that it can learn from datasets containing only low-resolution HSIs. With these contributions, our method is able to learn hyperspectral image super-resolution from heterogeneous datasets and lifts the requirement for having a large amount of high resolution (HR) HSI training samples. Extensive experiments on three standard datasets show that our method outperforms existing methods significantly and underpin the relevance of our contributions. Our code can be found at https://github.com/kli8996/HSISR.git.

show abstract

“…Typical examples include Crossstitch [15], Sluice [53] and NDDR [54]. Compared with the learning upon single modalities, multitask learning is not always beneficial as the performance is likely to be harmed by the negative transfer (negative knowledge transfer across tasks), which is clarified in [55], [56], [57]. [58] distills information across different tasks with multimodal feature aggregation.…”

Section: Deep Multimodal Fusionmentioning

confidence: 99%

“…[58] distills information across different tasks with multimodal feature aggregation. [57], [59] explicitly enforce cycle-based consistency between domains to improve performance and generalization. In this paper, we integrate the benefits of both hard parameter-sharing and soft parameter-sharing.…”

Section: Deep Multimodal Fusionmentioning

confidence: 99%

See 1 more Smart Citation

Channel Exchanging Networks for Multimodal and Multitask Dense Image Prediction

Wang¹,

Sun²,

Huang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Multimodal fusion and multitask learning are two vital topics in machine learning. Despite the fruitful progress, existing methods for both problems are still brittle to the same challenge-it remains dilemmatic to integrate the common information across modalities (resp. tasks) meanwhile preserving the specific patterns of each modality (resp. task). Besides, while they are actually closely related to each other, multimodal fusion and multitask learning are rarely explored within the same methodological framework before. In this paper, we propose Channel-Exchanging-Network (CEN) which is self-adaptive, parameter-free, and more importantly, applicable for both multimodal fusion and multitask learning. At its core, CEN dynamically exchanges channels between subnetworks of different modalities. Specifically, the channel exchanging process is self-guided by individual channel importance that is measured by the magnitude of Batch-Normalization (BN) scaling factor during training. For the application of dense image prediction, the validity of CEN is tested by four different scenarios: multimodal fusion, cycle multimodal fusion, multitask learning, and multimodal multitask learning. Extensive experiments on semantic segmentation via RGB-D data and image translation through multi-domain input verify the effectiveness of our CEN compared to current state-of-the-art methods. Detailed ablation studies have also been carried out, which provably affirm the advantage of each component we propose.

show abstract

Robust Learning Through Cross-Task Consistency

Cited by 109 publications

References 20 publications

Multi-Task Learning for Dense Prediction Tasks: A Survey

Multi-Task Learning for Dense Prediction Tasks: A Survey

Hyperspectral Image Super-Resolution with RGB Image Super-Resolution as an Auxiliary Task

Channel Exchanging Networks for Multimodal and Multitask Dense Image Prediction

Contact Info

Product

Resources

About