Unsupervised image-to-image translation is an important and challenging problem in computer vision. Given an image in the source domain, the goal is to learn the conditional distribution of corresponding images in the target domain, without seeing any examples of corresponding image pairs. While this conditional distribution is inherently multimodal, existing approaches make an overly simplified assumption, modeling it as a deterministic one-to-one mapping. As a result, they fail to generate diverse outputs from a given source domain image. To address this limitation, we propose a Multimodal Unsupervised Image-to-image Translation (MUNIT) framework. We assume that the image representation can be decomposed into a content code that is domain-invariant, and a style code that captures domain-specific properties. To translate an image to another domain, we recombine its content code with a random style code sampled from the style space of the target domain. We analyze the proposed framework and establish several theoretical results. Extensive experiments with comparisons to state-of-the-art approaches further demonstrate the advantage of the proposed framework. Moreover, our framework allows users to control the style of translation outputs by providing an example style image. Code and pretrained models are available at https://github.com/nvlabs/MUNIT.

sky sea tree cloud mountain grass Figure 1: Our model allows user control over both semantic and style as synthesizing an image. The semantic (e.g., the existence of a tree) is controlled via a label map (the top row), while the style is controlled via the reference style image (the leftmost column). Please visit our website for interactive image synthesis demos. AbstractWe propose spatially-adaptive normalization, a simple but effective layer for synthesizing photorealistic images given an input semantic layout. Previous methods directly feed the semantic layout as input to the deep network, which is then processed through stacks of convolution, normalization, and nonlinearity layers. We show that this is suboptimal as the normalization layers tend to "wash away" semantic information. To address the issue, we propose using the input layout for modulating the activations in normalization layers through a spatially-adaptive, learned transformation. Experiments on several challenging datasets demonstrate the advantage of the proposed method over existing approaches, regarding both visual fidelity and alignment with input layouts. Finally, our model allows user control over both semantic and style.

We present a compact but effective CNN model for optical flow, called PWC-Net. PWC-Net has been designed according to simple and well-established principles: pyramidal processing, warping, and the use of a cost volume. Cast in a learnable feature pyramid, PWC-Net uses the current optical flow estimate to warp the CNN features of the second image. It then uses the warped features and features of the first image to construct a cost volume, which is processed by a CNN to estimate the optical flow. PWC-Net is 17 times smaller in size and easier to train than the recent FlowNet2 model. Moreover, it outperforms all published optical flow methods on the MPI Sintel final pass and KITTI 2015 benchmarks, running at about 35 fps on Sintel resolution (1024×436) images. Our models are available on https://github.com/NVlabs/PWC-Net.

Although type 2 diabetes (T2D) is a major comorbidity of novel coronavirus disease 2019 (COVID-19), the impact of blood glucose control on the degree of medical interventions required and on all-cause mortality of patients with COVID-19 and pre-existing T2D remains unclear. Here, Zhu et al. report that among $7,300 individuals with COVID-19 (among which nearly 1,000 had T2D) in Hubei Province, China, those with T2D had significantly increased medical interventions and mortality risk. But among the patients with T2D, those with well-controlled blood glucose regulation (upper limit % 10 mmol/L) fared much better than those with poorly controlled blood glucose (upper limit > 10 mmol/L). These findings provide clinical evidence correlating more proper blood glucose control with improved outcomes in patients with COVID-19.

Rationale: Use of ACEIs (angiotensin-converting enzyme inhibitors) and ARBs (angiotensin II receptor blockers) is a major concern for clinicians treating coronavirus disease 2019 (COVID-19) in patients with hypertension. Objective: To determine the association between in-hospital use of ACEI/ARB and all-cause mortality in patients with hypertension and hospitalized due to COVID-19. Methods and Results: This retrospective, multi-center study included 1128 adult patients with hypertension diagnosed with COVID-19, including 188 taking ACEI/ARB (ACEI/ARB group; median age 64 [interquartile range, 55–68] years; 53.2% men) and 940 without using ACEI/ARB (non-ACEI/ARB group; median age 64 [interquartile range 57–69]; 53.5% men), who were admitted to 9 hospitals in Hubei Province, China from December 31, 2019 to February 20, 2020. In mixed-effect Cox model treating site as a random effect, after adjusting for age, gender, comorbidities, and in-hospital medications, the detected risk for all-cause mortality was lower in the ACEI/ARB group versus the non-ACEI/ARB group (adjusted hazard ratio, 0.42 [95% CI, 0.19–0.92]; P =0.03). In a propensity score-matched analysis followed by adjusting imbalanced variables in mixed-effect Cox model, the results consistently demonstrated lower risk of COVID-19 mortality in patients who received ACEI/ARB versus those who did not receive ACEI/ARB (adjusted hazard ratio, 0.37 [95% CI, 0.15–0.89]; P =0.03). Further subgroup propensity score-matched analysis indicated that, compared with use of other antihypertensive drugs, ACEI/ARB was also associated with decreased mortality (adjusted hazard ratio, 0.30 [95% CI, 0.12–0.70]; P =0.01) in patients with COVID-19 and coexisting hypertension. Conclusions: Among hospitalized patients with COVID-19 and coexisting hypertension, inpatient use of ACEI/ARB was associated with lower risk of all-cause mortality compared with ACEI/ARB nonusers. While study interpretation needs to consider the potential for residual confounders, it is unlikely that in-hospital use of ACEI/ARB was associated with an increased mortality risk.

Visual signals in a video can be divided into content and motion. While content specifies which objects are in the video, motion describes their dynamics. Based on this prior, we propose the Motion and Content decomposed Generative Adversarial Network (MoCoGAN) framework for video generation. The proposed framework generates a video by mapping a sequence of random vectors to a sequence of video frames. Each random vector consists of a content part and a motion part. While the content part is kept fixed, the motion part is realized as a stochastic process. To learn motion and content decomposition in an unsupervised manner, we introduce a novel adversarial learning scheme utilizing both image and video discriminators. Extensive experimental results on several challenging datasets with qualitative and quantitative comparison to the state-of-theart approaches, verify effectiveness of the proposed framework. In addition, we show that MoCoGAN allows one to generate videos with same content but different motion as well as videos with different content and same motion.

Unsupervised image-to-image translation methods learn to map images in a given class to an analogous image in a different class, drawing on unstructured (non-registered) datasets of images. While remarkably successful, current methods require access to many images in both source and destination classes at training time. We argue this greatly limits their use. Drawing inspiration from the human capability of picking up the essence of a novel object from a small number of examples and generalizing from there, we seek a few-shot, unsupervised image-to-image translation algorithm that works on previously unseen target classes that are specified, at test time, only by a few example images. Our model achieves this few-shot generation capability by coupling an adversarial training scheme with a novel network design. Through extensive experimental validation and comparisons to several baseline methods on benchmark datasets, we verify the effectiveness of the proposed framework. Our implementation and datasets are available at https://github.com/NVlabs/FUNIT.

No abstract