Multimodal Unsupervised Image-to-Image Translation

Huang, Xun; Li, Mingyu; Belongie, Serge; Kautz, Jan

doi:10.1007/978-3-030-01219-9_11

Cited by 1,948 publications

(2,381 citation statements)

References 57 publications

Supporting

Mentioning

2,375

Contrasting

Unclassified

Order By: Relevance

“…where z is the activation of the previous convolutional layer, which is first normalized. Then it is scaled by γ and shifted by β, which are parameters generated by a multilayer perceptron adapted from [10]. Compared to existing colorization models [2,34] that incorporate color conditions via a simple element-wise addition, AdaIN allows the model to produce vivid colorizations as shown in Fig.…”

Section: Colorization Networkmentioning

confidence: 99%

Coloring With Limited Data: Few-Shot Colorization via Memory Augmented Networks

Yoo¹,

Bahng²,

Chung³

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

128

View full text Add to dashboard Cite

Despite recent advancements in deep learning-based automatic colorization, they are still limited when it comes to few-shot learning. Existing models require a significant amount of training data. To tackle this issue, we present a novel memory-augmented colorization model Memo-Painter that can produce high-quality colorization with limited data. In particular, our model is able to capture rare instances and successfully colorize them. We also propose a novel threshold triplet loss that enables unsupervised training of memory networks without the need of class labels. Experiments show that our model has superior quality in both few-shot and one-shot colorization tasks.

show abstract

Section: Colorization Networkmentioning

confidence: 99%

Coloring With Limited Data: Few-Shot Colorization via Memory Augmented Networks

Yoo¹,

Bahng²,

Chung³

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

128

View full text Add to dashboard Cite

show abstract

“…For instance, the task of identifying the landscape in images can range from rivers, blue sky, freezing tundras to grasslands, forests, etc. Thus, some works [9,10] have proposed to generate effects such as different seasons artificially to augment the dataset. In this paper, we focus on recognizing natural objects like cat, dog, car, people, etc.…”

Section: Related Workmentioning

confidence: 99%

Attentive Cutmix: An Enhanced Data Augmentation Approach for Deep Learning Based Image Classification

Walawalkar

Shen

Liu

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Convolutional neural networks (CNN) are capable of learning robust representation with different regularization methods and activations as convolutional layers are spatially correlated. Based on this property, a large variety of regional dropout strategies have been proposed, such as Cutout [1], DropBlock [2], CutMix [3], etc. These methods aim to promote the network to generalize better by partially occluding the discriminative parts of objects. However, all of them perform this operation randomly, without capturing the most important region(s) within an object. In this paper, we propose Attentive CutMix, a naturally enhanced augmentation strategy based on CutMix [3]. In each training iteration, we choose the most descriptive regions based on the intermediate attention maps from a feature extractor, which enables searching for the most discriminative parts in an image. Our proposed method is simple yet effective, easy to implement and can boost the baseline significantly. Extensive experiments on CIFAR-10/100 datasets with various CNN architectures (in a unified setting) demonstrate the effectiveness of our proposed method, which consistently outperforms the baseline CutMix and other methods by a significant margin.

show abstract

“…The D s also has 4 residual blocks which is similar to E c i , except that the feature map size is upsampled by 2 with channel number halved after each stage. For image reconstruction, we generally follow the practice in [7]. Specifically, each E a i consists of 5 convolutional layers followed by a global average pooling and a fully-connected layer to obtain the appearance code.…”

Section: Learning Process and Network Architecturementioning

confidence: 99%

Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion

Chen

Dou

Jin

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Accurate medical image segmentation commonly requires effective learning of the complementary information from multimodal data. However, in clinical practice, we often encounter the problem of missing imaging modalities. We tackle this challenge and propose a novel multimodal segmentation framework which is robust to the absence of imaging modalities. Our network uses feature disentanglement to decompose the input modalities into the modality-specific appearance code, which uniquely sticks to each modality, and the modality-invariant content code, which absorbs multimodal information for the segmentation task. With enhanced modality-invariance, the disentangled content code from each modality is fused into a shared representation which gains robustness to missing data. The fusion is achieved via a learning-based strategy to gate the contribution of different modalities at different locations. We validate our method on the important yet challenging multimodal brain tumor segmentation task with the BRATS challenge dataset. With competitive performance to the state-of-the-art approaches for full modality, our method achieves outstanding robustness under various missing modality(ies) situations, significantly exceeding the state-of-the-art method by over 16% in average for Dice on whole tumor segmentation.

show abstract

Multimodal Unsupervised Image-to-Image Translation

Cited by 1,948 publications

References 57 publications

Coloring With Limited Data: Few-Shot Colorization via Memory Augmented Networks

Coloring With Limited Data: Few-Shot Colorization via Memory Augmented Networks

Attentive Cutmix: An Enhanced Data Augmentation Approach for Deep Learning Based Image Classification

Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion

Contact Info

Product

Resources

About