Learning Correspondence From the Cycle-Consistency of Time

Wang, Xiaolong; Jabri, Allan; Efros, Alexei A.

doi:10.1109/cvpr.2019.00267

Cited by 451 publications

(380 citation statements)

References 78 publications

Supporting

Mentioning

379

Contrasting

Order By: Relevance

“…1, we propose CycleVAE, which is capable of recycling the converted spectra back into the system, so that the conversion flow is indirectly considered in the parameter optimization. A similar idea has also been proposed as a cycle-consistent flow in a self-supervised method for visual correspondence [24].…”

Section: Proposed Cyclevae-based Vcmentioning

confidence: 99%

“…In this paper, to improve VAE-based VC, we propose to use cycle-consistent mapping flow [24], i.e., CycleVAE-based VC, that indirectly optimizes the conversion flow by recycling the converted spectral features. Specifically, in the proposed CycleVAE, the converted features are fed-back into the system to generate corresponding cyclic reconstructed spectra that can be directly optimized.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Non-Parallel Voice Conversion with Cyclic Variational Autoencoder

Tobing

Hayashi

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

In this paper, we present a novel technique for a non-parallel voice conversion (VC) with the use of cyclic variational autoencoder (CycleVAE)-based spectral modeling. In a variational autoencoder (VAE) framework, a latent space, usually with a Gaussian prior, is used to encode a set of input features. In a VAE-based VC, the encoded latent features are fed into a decoder, along with speaker-coding features, to generate estimated spectra with either the original speaker identity (reconstructed) or another speaker identity (converted). Due to the non-parallel modeling condition, the converted spectra can not be directly optimized, which heavily degrades the performance of a VAEbased VC. In this work, to overcome this problem, we propose to use CycleVAE-based spectral model that indirectly optimizes the conversion flow by recycling the converted features back into the system to obtain corresponding cyclic reconstructed spectra that can be directly optimized. The cyclic flow can be continued by using the cyclic reconstructed features as input for the next cycle. The experimental results demonstrate the effectiveness of the proposed CycleVAE-based VC, which yields higher accuracy of converted spectra, generates latent features with higher correlation degree, and significantly improves the quality and conversion accuracy of the converted speech.

show abstract

Section: Proposed Cyclevae-based Vcmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Non-Parallel Voice Conversion with Cyclic Variational Autoencoder

Tobing

Hayashi

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

show abstract

“…Li et al [19] proposed a framework with adaptive feature propagation for high-level features to reduce the latency of video semantic segmentation. Wang et al [20] used an unsupervised method to learn feature representations for identifying correspondences across frames. Lee et al [21] attempted to derive semantic correspondences by objectaware losses.…”

Section: B Semantics Sharingmentioning

confidence: 99%

Boosting Real-Time Driving Scene Parsing With Shared Semantics

Xiang

Bao

et al. 2020

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Real-time scene parsing is a fundamental feature for autonomous driving vehicles with multiple cameras. In this letter we demonstrate that sharing semantics between cameras with different perspectives and overlapped views can boost the parsing performance when compared with traditional methods, which individually process the frames from each camera. Our framework is based on a deep neural network for semantic segmentation but with two kinds of additional modules for sharing and fusing semantics. On the one hand, a semantics sharing module is designed to establish the pixel-wise mapping between the input images. Features as well as semantics are shared by the map to reduce duplicated workload which leads to more efficient computation. On the other hand, feature fusion modules are designed to combine different modal of semantic features, which leverage the information from both inputs for better accuracy. To evaluate the effectiveness of the proposed framework, we have applied our network to a dual-camera vision system for driving scene parsing. Experimental results show that our network outperforms the baseline method on the parsing accuracy with comparable computations.

show abstract

“…It purely uses the input data to create auxiliary tasks and enables deep networks to learn effective latent features by solving these auxiliary tasks. Various strategies have been proposed to construct auxiliary tasks, based on temporal correspondence (Li et al, 2019b;Wang et al, 2019a), cross-modal consistency , etc. In computer vision, examples of auxiliary tasks include rotation prediction (Gidaris et al, 2018a), image inpainting (Pathak et al, 2016a), automatic colorization (Zhang et al, 2016b), instance discrimination (Wu et al, 2018), to name a few.…”

Section: Introductionmentioning

confidence: 99%