SUPERB: Speech processing Universal PERformance Benchmark

Yang, Shu-Wen; Chi, Po-Han; Chuang, Yung-Sung; Lai, Cheng-I; Lakhotia, Kushal; Lin, Yist Y.; Liu, Andy T.; Shi, Jiatong; Chang, Xuankai; Lin, Gwo‐Fong; Huang, Tsun‐Sheng; Tseng, Wei-Cheng; Lee, Ko-tik; Liu, Da-Rong; Huang, Zili; Dong, Shuyan; Li, Shang-Wen; Watanabe, Shinji; Mohamed, Abdelrahman; Lee, Hung-yi

doi:10.48550/arxiv.2105.01051

Cited by 30 publications

(38 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the three benchmarks Speech Commands V1, Vox-Celeb1, and IEMOCAP that the original AST has not been tested on, we use the standard SUPERB (Yang et al 2021) training and testing framework. Specifically, we search the learning rate from 1e-5 to 1e-3 for out SSAST model and all baseline models and train the model for up to 20k, 40k, and 10k iterations for Speech Commands V2, VocCeleb1, and IEMOCAP, respectively.…”

Section: Downstream Fine-tuning Detailsmentioning

confidence: 99%

SSAST: Self-Supervised Audio Spectrogram Transformer

Yang¹,

Lai²,

Chung³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recently, neural networks based purely on self-attention, such as the Vision Transformer (ViT), have been shown to outperform deep learning models constructed with convolutional neural networks (CNNs) on various vision tasks, thus extending the success of Transformers, which were originally developed for language processing, to the vision domain. A recent study (Gong, Chung, and Glass 2021) showed that a similar methodology can also be applied to the audio domain. Specifically, the Audio Spectrogram Transformer (AST) achieves state-of-the-art results on various audio classification benchmarks. However, pure Transformer models tend to require more training data compared to CNNs, and the success of the AST relies on supervised pretraining that requires a large amount of labeled data and a complex training pipeline, thus limiting the practical usage of AST. This paper focuses on audio and speech classification, and aims to alleviate the data requirement issues with the AST by leveraging self-supervised learning using unlabeled data. Specifically, we propose to pretrain the AST model with joint discriminative and generative masked spectrogram patch modeling (MSPM) using unlabeled audio from AudioSet and Librispeech. We evaluate our pretrained models on both audio and speech classification tasks including audio event classification, keyword spotting, emotion recognition, and speaker identification. The proposed self-supervised framework significantly boosts AST performance on all tasks, with an average improvement of 60.9%, leading to similar or even better results than a supervised pretrained AST. To the best of our knowledge, it is the first patch-based self-supervised learning framework in the audio and speech domain, and also the first self-supervised learning framework for AST.

show abstract

Section: Downstream Fine-tuning Detailsmentioning

confidence: 99%

SSAST: Self-Supervised Audio Spectrogram Transformer

Yang¹,

Lai²,

Chung³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…HuBERT is a self-supervised learning based model trained with masked continuous audio signals. It shows superior performance across multiple tasks such as ASR, spoken language modeling, and speech synthesis [20]. For the HuBERT model, we use the k-means algorithm to cluster the extracted latent representations into the clustering center while the VQ-VAE directly use the nearest centroid in VQ-codebook .…”

Section: Discretization Of Speechmentioning

confidence: 99%

“…In preliminary experiments, VQ-VAE was used with the same architecture proposed by [25] for the VCTK corpus. Following that, inspired by the impressive performance with HuBERT discrete [20], we also use the HiFi-GAN to synthesize the speech signal given the HuBERT discrete symbols from the pre-trained model [29]. For all experiments, we use the HuBERT Large model trained on Libri-Light [30] 60k hour without any downstream finetuning to extract the features.…”

Section: Discrete Vocodermentioning

confidence: 99%

Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem

Shi¹,

Chang²,

Tomoki³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Deep learning based models have significantly improved the performance of speech separation with input mixtures like the cocktail party. Prominent methods (e.g., frequency-domain and time-domain speech separation) usually build regression models to predict the ground-truth speech from the mixture, using the masking-based design and the signal-level loss criterion (e.g., MSE or SI-SNR). This study demonstrates, for the first time, that the synthesis-based approach can also perform well on this problem, with great flexibility and strong potential. Specifically, we propose a novel speech separation/enhancement model based on the recognition of discrete symbols, and convert the paradigm of the speech separation/enhancement related tasks from regression to classification. By utilizing the synthesis model with the input of discrete symbols, after the prediction of discrete symbol sequence, each target speech could be resynthesized. Evaluation results based on the WSJ0-2mix and VCTKnoisy corpora in various settings show that our proposed method can steadily synthesize the separated speech with high speech quality and without any interference, which is difficult to avoid in regressionbased methods. In addition, with negligible loss of listening quality, the speaker conversion of enhanced/separated speech could be easily realized through our method.

show abstract

“…With the popularization of self-supervised learning (SSL), the speech community has made significant progress towards reducing the amount of labelled data needed to reach strong levels of performance in a variety of tasks. To better measure the progress within this research area, two benchmarks, the ZeroSpeech 2021 challenge (Nguyen et al 2020;Alishahi et al 2021) and the SUPERB benchmark (Yang et al 2021), were recently proposed.…”

Section: Introductionmentioning

confidence: 99%

Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling

Puyuan¹,

Harwath²

2022

Preprint

View full text Add to dashboard Cite

In this paper, we describe our submissions to the ZeroSpeech 2021 Challenge and SUPERB benchmark. Our submissions are based on the recently proposed FaST-VGS model, which is a Transformer-based model that learns to associate raw speech waveforms with semantically related images, all without the use of any transcriptions of the speech. Additionally, we introduce a novel extension of this model, FaST-VGS+ 1 , which is learned in a multi-task fashion with a masked language modeling objective in addition to the visual grounding objective. On ZeroSpeech 2021, we show that our models perform competitively on the ABX task, outperform all other concurrent submissions on the Syntactic and Semantic tasks, and nearly match the best system on the Lexical task. On the SUPERB benchmark, we show that our models also achieve strong performance, in some cases even outperforming the popular wav2vec2.0 model.

show abstract

SUPERB: Speech processing Universal PERformance Benchmark

Cited by 30 publications

References 38 publications

SSAST: Self-Supervised Audio Spectrogram Transformer

SSAST: Self-Supervised Audio Spectrogram Transformer

Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem

Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling

Contact Info

Product

Resources

About