Multimodal Fusion of Multirate Acoustic, Prosodic, and Lexical Speaker Characteristics for Native Language Identification

Abstract

Native language identification from acoustic signals of L2 speakers can be useful in a range of applications such as informing automatic speech recognition (ASR), speaker recognition, and speech biometrics. In this paper we follow a multistream and multi-rate approach, for native language identification, in feature extraction, classification, and fusion. On the feature front we employ acoustic features such as MFCC and PLP features, at different time scales and different transformations; we evaluate speaker normalization as a feature and as a transform; investigate phonemic confusability and its interplay with paralinguistic cues at both the frame and phone-level temporal scales; and automatically extract lexical features; in addition to baseline features. On the classification side we employ SVM, i-Vector, DNN and bottleneck features, and maximum-likelihood models. Finally we employ fusion for system combination and analyze the complementarity of the individual systems. Our proposed system significantly outperforms the baseline system on both development and test sets.

Improving Sub-Phone Modeling for Better Native Language Identification with Non-Native English Speech

“…The most confusable classification are between Hindi and Telugu, which are both languages used in India. Similar observations were found in the systems reported in [11,12], although with a lower count.…”

“…It indicates that 600-dim i-Vector extracted from the posterior supervector of GMM with 1,024 Gaussian components can achieve the best performance. These results are similar to those that the Challenge systems [11,12] obtained, i.e., approximately 76% for both UAR and Acc by only using corpus, NNSE, provided by ComParE organizers. Thereafter, the dimension of i-vector is fixed to 600.…”

“…Our system with TDNN based i-Vector trained by Librispeech can achieve 82.9% UAR, which is 1.6% better than the L 2 F system which fuses two i-vector based systems. In [12], Librispeech is also used to train a phone recognizer for extracting L1pronunciation projection features which are used for i-vector extraction. The fusion system achieves 79.9% Acc and 80.1% UAR.…”

“…This PLLR based i-vector system is fused together with the MFCC feature based i-vector system at the scoring level and achieves the best performance in ComParE. Various features at the frame, phone and lexical levels, multiple classifiers, SVM, PLDA and DNN, and fusion at both feature and score levels are investigated in detail [12]. The performance of the final fused system is just slightly worse than the best system.…”

Applying Speech Tempo-Derived Features, BoAW and Fisher Vectors to Detect Elderly Emotion and Speech in Surgical Masks

“…For example, Grèzes et al calculated the ratio of speaker overlap to aid conflict intensity estimation [13]; Montacié and Caraty detected temporal events (e.g. speech onset latency, event starting time-codes, pause and phone segments) to detect cognitive load [14], several authors extracted phone posterior-based attributes to determine the degree of nativeness or the native language of the speaker [15,16,17], while Huckvale and Beke developed specific spectral-based attributes to detect whether the speaker has a cold [18]. Of course, some kind of fusion of the general and the task-specific attributes might also prove to be beneficial.…”