An End-to-End Approach to Automatic Speech Assessment for Cantonese-speaking People with Aphasia

Abstract

Conventional automatic assessment of pathological speech usually follows two main steps: (1) extraction of pathology-specific features; (2) classification or regression on extracted features. Given the great variety of speech and language disorders, feature design is never a straightforward task, and yet it is most crucial to the performance of assessment. This paper presents an end-to-end approach to automatic speech assessment for Cantonese-speaking People With Aphasia (PWA). The assessment is formulated as a binary classification task to discriminate PWA with high scores of subjective assessment from those with low scores. The sequence-to-one Recurrent Neural Network with Gated Recurrent Unit (GRU-RNN) and Convolutional Neural Network (CNN) models are applied to realize the end-to-end mapping from fundamental speech features to the classification result. The pathology-specific features used for assessment can be learned implicitly by the neural network model. Class Activation Mapping (CAM) method is utilized to visualize how those features contribute to the assessment result. Our experimental results 2 Ying Qin et al.show that the end-to-end approach outperforms the conventional two-step approach in the classification task, and confirm that the CNN model is able to learn impairment-related features that are similar to human-designed features. The experimental results also suggest that CNN model performs better than sequence-to-one GRU-RNN model in this specific task.

Automatic Assessment of Language Impairment Based on Raw ASR Output

“…The CNs is effective to improve robustness of embedding representations of ASR output and thus improving the assessment performance. We follow the approach in [11] to develop an end-to-end assessment system. With the same arrangement of 5-fold cross validation on 92 impaired speakers, a CNN model with global average pooling is trained with Log-Mel acoustic features from PWA speech.…”

“…Generally speaking, use of expert knowledge in feature design and extraction played an important role in most previous studies on automatic assessment of language impairment. Recently, we attempted an end-to-end approach to the problem, in which a direct mapping from conventional acoustic features (i.e., Log-Mel filter-bank features) to assessment score was realized with a Convolutional Neural Network (CNN) model [11]. By visualizing the learned features using class activation mapping [12], it was confirmed that the CNN model could capture impairment-related acoustic characteristics that are in agreement with human-designed features.…”

Performance Evaluation of Machine Learning Frameworks for Aphasia Assessment

“…Speech assessment is used extensively in the diagnosis of Parkinson’s and aphasia diseases [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 ]. Aphasia is an acquired neurogenic language disorder that can be evaluated with one of the well-known assessment tools, such as the Chinese Rehabilitation Research Center Aphasia Examination (CRRCAE [ 15 ], for Chinese-dialect-speaking patients), the Aachen Aphasia Test (AAT [ 16 ], for German-speaking patients) and the Boston Diagnostic Aphasia Examination (BDAE [ 17 ], for English-speaking patients).…”

“…Commonly, there are three aphasia assessment tasks whereby an SLP performs a comprehensive examination of the patient’s communication abilities, including speaking, expressing ideas, understanding language, reading, and writing. These tasks are the discrimination between normal and aphasic speech [ 9 ], the assessment of the degree of severity of impairment for aphasic patients [ 10 , 12 ], and the classification of aphasia syndromes (such as Global aphasia, Broca’s aphasia, Wernicke’s aphasia and amnesic aphasia) [ 13 , 14 ]. Conventional methods of aphasia assessment and rehabilitation are resource-intensive processes that require the presence of an SLP.…”

“…Most research on automatic aphasia assessment [ 5 , 9 , 10 , 12 , 13 , 14 , 18 ] has focused on a single aphasia assessment task using automatic speech recognition techniques and a machine learning (ML) framework with a fixed dataset-type . Generally, three types of training datasets are used for aphasia assessment tasks: (1) healthy subjects’ dataset, (2) aphasic patients’ dataset, or (3) a combination of healthy and aphasic dataset [ 10 , 12 ] (hereafter, these datasets are referred to as dataset-type .…”