From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback Constraint

Cai, Zexin; Zhang, Chuxiong; Li, Ming

doi:10.48550/arxiv.2005.04587

Cited by 8 publications

(11 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In a typical multi-speaker TTS system [54], speaker information can be provided by the speaker embedding extracted by a speaker encoder that is trained in a speaker verification task with speech as input. Here, we also trained a speech-based speaker encoder in the speaker verification task with the large margin softmax loss [55].…”

Section: B Face Encodermentioning

confidence: 99%

“…As mentioned in the last subsection, due to the inaccessible of speech-face paired database, we can not train a faceconditioned multi-speaker TTS directly. Instead, we train a multi-speaker TTS model with text-speech paired database as that in general multi-speaker TTS method [54]. Specifically, during the training process, speaker information can be provided by the speaker embedding extracted with a speaker encoder.…”

Section: Face-conditioned Multi-speaker Ttsmentioning

confidence: 99%

“…Following the speaker embedding based multi-speaker TTS model [54], the speaker embedding in the training phase is engaged after the CBHG block. It works as a speaker attribute to provide speaker information for the TTS system.…”

Section: Face-conditioned Multi-speaker Ttsmentioning

confidence: 99%

See 2 more Smart Citations

AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person

Wang,

Xie,

Zhu

et al. 2021

Preprint

View full text Add to dashboard Cite

Automatically generating videos in which synthesized speech is synchronized with lip movements in a talking head has great potential in many human-computer interaction scenarios. In this paper, we present an automatic method to generate synchronized speech and talking-head videos on the basis of text and a single face image of an arbitrary person as input. In contrast to previous text-driven talking head generation methods, which can only synthesize the voice of a specific person, the proposed method is capable of synthesizing speech for any person that are inaccessible in the training stage. Specifically, the proposed method decomposes the generation of synchronized speech and talking head videos into two stages, i.e., a text-tospeech (TTS) stage and a speech-driven talking head generation stage. The proposed TTS module is a face-conditioned multispeaker TTS model that gets the speaker identity information from face images instead of speech, which allows us to synthesize a personalized voice on the basis of the input face image. To generate the talking head videos from the face images, a facial landmark-based method that can predict both lip movements and head rotations is proposed. Extensive experiments demonstrate that the proposed method is able to generate synchronized speech and talking head videos for arbitrary persons and nonpersons. Synthesized speech shows consistency with the given face regarding to the synthesized voice's timbre and one's appearance in the image, and the proposed landmark-based talking head method outperforms the state-of-the-art landmark-based method on generating natural talking head videos.

show abstract

Section: B Face Encodermentioning

confidence: 99%

Section: Face-conditioned Multi-speaker Ttsmentioning

confidence: 99%

See 1 more Smart Citation

AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person

Wang,

Xie,

Zhu

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…We also apply the loss proposed by [16] to the output of PostNet as the speaker reconstruction loss. Denoted the speaker embedding extracted from target Mel-spectrogram as s and the one extracted from the predicted Mel-spectrogram as ŝ, then the speaker reconstruction loss is…”

Section: Supervision On Prediceted Mel-spectrogrammentioning

confidence: 99%

“…Then we erase the speaker information from the intermediate representation by incorporating the speaker encoder. Our proposed system also applies the feedback loss control for speaker embedding, which is proposed by [16] and applied on VC in [17]. By adding this loss, the generated results of our proposed system demonstrate high spoofing capability to speaker verification systems.…”

Section: Introductionmentioning

confidence: 98%

SIG-VC: A Speaker Information Guided Zero-shot Voice Conversion System for Both Human Beings and Machines

Zhang¹,

Cai²,

Qin³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Nowadays, as more and more systems achieve good performance in traditional voice conversion (VC) tasks, people's attention gradually turns to VC tasks under extreme conditions. In this paper, we propose a novel method for zero-shot voice conversion. We aim to obtain intermediate representations for speaker-content disentanglement of speech to better remove speaker information and get pure content information. Accordingly, our proposed framework contains a module that removes the speaker information from the acoustic feature of the source speaker. Moreover, speaker information control is added to our system to maintain the voice cloning performance. The proposed system is evaluated by subjective and objective metrics. Results show that our proposed system significantly reduces the trade-off problem in zero-shot voice conversion, while it also manages to have high spoofing power to the speaker verification system.

show abstract

Single-chip multi-processing architecture for spaceborne SAR imaging and intelligent processing

Wang

Zhang

Huang

et al. 2021

JNWPU

View full text Add to dashboard Cite

The satellite-borne SAR image intelligent processing system needs to process on-orbit real-time imaging and various tasks of applications, for which reason designing a dedicated high-efficient single-chip multi-processor is of prioritized necessity that can simultaneously satisfy requirements of real-time and low power consumption. Aiming at on-chip data organization and memory access structure, two typical models of SAR(synthetic aperture radar) imaging CSA (chirp scaling) and neural network VGG-11 are analyzed, and then a collaborative computing model for the intelligent processing on remote sensing is extracted. A strip Tile data processing scheme and a dedicated multi-processing architecture is not only proposed, but a data organization and a caching strategy of Tile space synchronization splicing is also presented. In addition, the designed data caching structure among the processing units greatly reduces off-chip access memory bandwidth while supporting parallel pipeline execution of multi-task model. The chip adopts 28 nm CMOS technology featuring with merely 1.83 W of the overall power consumption, whose throughput and energy efficiency reaches 9.89TOPS and 5.4 TOPS/W, respectively. The present architecture can improve real-time performance of the on-orbit remote sensing intelligent processing platform while reducing the complexity of system designing, which is highly adaptive to differentiated expansions according to different models of algorithm.

show abstract

From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback Constraint

Cited by 8 publications

References 16 publications

AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person

AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person

SIG-VC: A Speaker Information Guided Zero-shot Voice Conversion System for Both Human Beings and Machines

Single-chip multi-processing architecture for spaceborne SAR imaging and intelligent processing

Contact Info

Product

Resources

About