Sibilant consonants classification comparison with multi‐ and single‐class neural networks

Abstract

Many children with speech sound disorders cannot pronounce the sibilant consonants correctly. We have developed a serious game, which is controlled by the children's voices in real time, with the purpose of helping children on practicing the production of European Portuguese (EP) sibilant consonants. For this, the game uses a sibilant consonant classifier. Since the game does not require any type of adult supervision, children can practice producing these sounds more often, which may lead to faster improvements of their speech. Recently, the use of deep neural networks has given considerable improvements in the classification of a variety of use cases, from image classification to speech and language processing. Here, we propose to use deep convolutional neural networks to classify sibilant phonemes of EP in our serious game for speech and language therapy. We compared the performance of several different artificial neural networks that used Mel frequency cepstral coefficients or log Mel filterbanks. Our best deep learning model achieves classification scores of 95.48% using a 2D convolutional model with log Mel filterbanks as input features. Such results are then further improved for specific classes with simple binary classifiers.

The BioVisualSpeech Corpus of Words with Sibilants for Speech Therapy Games Development

“…The 1D convolutional model was able to increase the score to 94.04%. The 2D convolutional model with log Mel filterbanks achieved a classification score of 95.48% [26]. This could be expected since the convolutional layers can extract localized information that can contribute to a better classification than that of simpler ANN models.…”

“…The BioVisualSpeech isolated sibilants game was designed to help children train the production of the four EP sibilant consonants [25,26]. This game implements the isolated sibilant therapy exercise.…”

“…Other models included different neural network models, from simple hidden-layers artificial neural networks (ANN) to more complex 1D and 2D CNNs that used either MFCC or log Mel filterbanks as input [26]. We used the Adam optimizer as our loss function in all networks and we also used the stochastic gradient descent (SGD) in our CNNs [27].…”

“…Since the models have high classification scores and low false negative rates, they are suitable for use in speech and language therapy games for sigmatism. For a detailed evaluation of the different four class sibilant classifier models, please check reference [26].…”

Fifth special issue on knowledge discovery and business intelligence

“…Therefore, the adapted W-Net model can be used as a valuable image processing tool, alleviating the need for a manual inspection of aerial images. Another deep learning model, the Convolutional Neural Network (CNN), was adapted by Anjos, Marques, and Grilo (2020) to classify sibilant phonemes of European Portuguese sounds. The paper, entitled "Sibilant consonants classification comparison with multi and single-class neural networks," proposes a serious sound game, aiming to help children to improve their speech.…”