Automated Mandarin tone classification using deep neural networks trained on a large speech dataset

Abstract

A tonal language is one in which the speaker’s intonation modifies the meaning of a word. In this work, we perform a rigorous analysis of intonation changes, or pitch contours, produced by native Mandarin speakers to predict the tone-contour type. Pitch contours are estimated using a number of different methods, also measuring each contour’s Mel-Frequency Cepstral Coefficients (MFCCs). The dataset used was autonomously generated from the Aishell open-source Mandarin speech corpus. Each sample was aligned with its transcript using Montreal Forced Alignment and segmented into individual words. The resulting corpus covers 11 topic domains, spoken by 400 individuals. Separate development, training, and testing datasets are created to ensure the integrity of our results. Pitch contours and their MFCCs are exposed to a number of machine learning techniques including clustered, regression, and traditional Deep Neural Network (DNN) approaches. MFCCs are additionally processed using convolutional neural networks. The models are used to predict the corresponding tone for a contour. Our work seeks to determine which intonation representations perform optimally for machine learning tasks. The tool is used to provide audio and visual feedback to learners of tonal languages. [Work supported by RPI Seed Grant and CISL].

The Application Analysis of Neural Network Techniques on Lexical Tone Rehabilitation of Mandarin-Speaking Patients With Post-Stroke Dysarthria

“…While the focus of the present study was individual tone production, that model was not suitable in our case. More recently, newly developed artificial intelligence methods were used to classify Mandarin tones [32], [34]. Chen et al used a CNN model based on MFCC to classify tones that produced by normal-hearing children.…”

“…Several previous studies also showed that ANNs were able to classify tones well in Mandarin Chinese [28]- [31]. Convolutional neural network (CNN) with machine learning algorithm was also adopted in recent years to classify Mandarin tones [32]- [34]. These newly developed neural networks performed even better comparing to the conventional ANN, with the reported accuracies of around 95% correctness.…”