Improving Shape Deformation in Unsupervised Image-to-Image Translation

Gokaslan, Aaron; Ramanujan, Vivek; Ritchie, Daniel; Kim, Kwang In; Tompkin, James

doi:10.48550/arxiv.1808.04325

Cited by 3 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Intriguingly, this offers an explanation for a number of rather disconnected findings: CNNs match texture appearance for humans (Wallis et al, 2017), and their predictive power for neural responses along the human ventral stream appears to be largely due to human-like texture representations, but not human-like contour representations (Laskar et al, 2018;Long & Konkle, 2018). Furthermore, texture-based generative modelling approaches such as style transfer (Gatys et al, 2016), single image super-resolution (Gondal et al, 2018) as well as static and dynamic texture synthesis (Gatys et al, 2015; all produce excellent results using standard CNNs, while CNNbased shape transfer seems to be very difficult (Gokaslan et al, 2018). CNNs can still recognise images with scrambled shapes Brendel & Bethge, 2019), but they have much more difficulties recognising objects with missing texture information (Ballester & de Araújo, 2016;Yu et al, 2017).…”

Section: Discussionmentioning

confidence: 99%

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

Geirhos¹,

Rubisch²,

Michaelis³

et al. 2018

Preprint

317

386

View full text Add to dashboard Cite

Convolutional Neural Networks (CNNs) are commonly thought to recognise objects by learning increasingly complex representations of object shapes. Some recent studies suggest a more important role of image textures. We here put these conflicting hypotheses to a quantitative test by evaluating CNNs and human observers on images with a texture-shape cue conflict. We show that ImageNettrained CNNs are strongly biased towards recognising textures rather than shapes, which is in stark contrast to human behavioural evidence and reveals fundamentally different classification strategies. We then demonstrate that the same standard architecture (ResNet-50) that learns a texture-based representation on ImageNet is able to learn a shape-based representation instead when trained on 'Stylized-ImageNet', a stylized version of ImageNet. This provides a much better fit for human behavioural performance in our well-controlled psychophysical lab setting (nine experiments totalling 48,560 psychophysical trials across 97 observers) and comes with a number of unexpected emergent benefits such as improved object detection performance and previously unseen robustness towards a wide range of image distortions, highlighting advantages of a shape-based representation.

show abstract

Section: Discussionmentioning

confidence: 99%

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

Geirhos¹,

Rubisch²,

Michaelis³

et al. 2018

Preprint

317

386

View full text Add to dashboard Cite

show abstract

Section: Normalizing Methodsmentioning

confidence: 99%

“…To overcome image deformation in unsupervised image translation, we combined the position-based selection strategy (Yang et al 2018) with multi-scale structure similarity loss (Wang et al 2003, Gokaslan et al 2018. We added skip connection to the generator to eliminate artifacts of the generated images by the generator (Gokaslan et al 2018). As new images of domain T could not have exactly the same characteristics as the training images of domain T, we also input new images of domain T into the generator to obtain a good consistency between all normalized images.…”

Section: Normalizing Methodsmentioning

confidence: 99%

Normalization of multicenter CT radiomics by a generative adversarial network method

Han

et al. 2021

Phys. Med. Biol.

View full text Add to dashboard Cite

To reduce the variability of radiomics features caused by computed tomography (CT) imaging protocols through using a generative adversarial network (GAN) method. In this study, we defined a set of images acquired with a certain imaging protocol as a domain, and a total of four domains (A, B, C, and T [target]) from three different scanners was included. In data set#1, 60 patients for each domain were collected. Data sets#2 and #3 included 40 slices of spleen for each of the domains. In data set#4, the slices of three colorectal cancer groups (n = 28, 38 and 32) were separately retrieved from three different scanners, and each group contained short-term and long-term survivors. Seventy-seven features were extracted for evaluation by comparing the feature distributions. First, we trained the GAN model on data set#1 to learn how to normalize images from domains A, B and C to T. Next, by comparing feature distributions between normalized images of the different domains, we identified the appropriate model and assessed it, in data set#2 and data set#3, respectively. Finally, to investigate whether our proposed method could facilitate multicenter radiomics analysis, we built the least absolute shrinkage and selection operator classifier to distinguish short-term from long-term survivors based on a certain group in data set#4, and validate it in another two groups, which formed a cross-validation between groups in data set#4. After normalization, the percentage of aligned features between domains A versus T, B versus T, and C versus T increased from 10.4 %, 18.2% and 50.1% to 93.5%, 89.6% and 77.9%, respectively. In the cross-validation results, the average improvement of the area under the receiver operating characteristic curve achieved 11% (3%–32%). Our proposed GAN-based normalization method could reduce the variability of radiomics features caused by different CT imaging protocols and facilitate multicenter radiomics analysis.

show abstract

“…In particular, the unpaired (or unsupervised) image-to-image translation has achieved an impressive progress based on variants of generative adversarial networks (GANs) Liu et al, 2017;Choi et al, 2017;Almahairi et al, 2018;Lee et al, 2018), and has also drawn considerable attention due to its practical applications including colorization , super-resolution (Ledig et al, 2017), semantic manipulation (Wang et al, 2018b), and domain adaptation (Bousmalis et al, 2017;Shrivastava et al, 2017;Hoffman et al, 2017). Previous methods on this line of research, however, often fail on challenging tasks, in particular, when the translation task involves significant changes in shape of instances or the images to translate contains multiple target instances (Gokaslan et al, 2018). Our goal is to extend image-to-image translation towards such challenging tasks, which can strengthen its applicability up to the next level, e.g., changing pants to skirts in fashion images for a customer to decide which one is better to buy.…”

Section: Introductionmentioning

confidence: 99%

“…To the best of our knowledge, we are the first to report image-to-image translation results for multiinstance transfiguration tasks. A few number of recent methods Liu et al, 2017;Gokaslan et al, 2018) show some transfiguration results but only for images with a single instance often in a clear background. Unlike the previous results in a simple setting, our focus is on the harmony of instances naturally rendered with the background.…”

Section: Introductionmentioning

confidence: 99%

InstaGAN: Instance-aware Image-to-Image Translation

Mo,

Cho,

Shin

2018

Preprint

View full text Add to dashboard Cite

Unsupervised image-to-image translation has gained considerable attention due to the recent impressive progress based on generative adversarial networks (GANs). However, previous methods often fail in challenging cases, in particular, when an image has multiple target instances and a translation task involves significant changes in shape, e.g., translating pants to skirts in fashion images. To tackle the issues, we propose a novel method, coined instance-aware GAN (InstaGAN), that incorporates the instance information (e.g., object segmentation masks) and improves multi-instance transfiguration. The proposed method translates both an image and the corresponding set of instance attributes while maintaining the permutation invariance property of the instances. To this end, we introduce a context preserving loss that encourages the network to learn the identity function outside of target instances. We also propose a sequential mini-batch inference/training technique that handles multiple instances with a limited GPU memory and enhances the network to generalize better for multiple instances. Our comparative evaluation demonstrates the effectiveness of the proposed method on different image datasets, in particular, in the aforementioned challenging cases. Code and results are available in https://github.com/sangwoomo/instagan.

show abstract

Improving Shape Deformation in Unsupervised Image-to-Image Translation

Cited by 3 publications

References 0 publications

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

Normalization of multicenter CT radiomics by a generative adversarial network method

InstaGAN: Instance-aware Image-to-Image Translation

Contact Info

Product

Resources

About