Towards Interpreting Deep Learning Models to Understand Loss of Speech Intelligibility in Speech Disorders Step 2: Contribution of the Emergence of Phonetic Traits

Abstract

Apart from the impressive performance it has achieved in several tasks, one of the most important factors remaining for the continuous progress of deep learning is the increased work related to interpretability, especially in a medical context. In a recent work, we presented competitive performance achieved with a CNN-based model trained on normal speech for the French phone classification and how it correlates well with different perceptual measures when exposed to disordered speech. This paper extends that work by focusing on interpretability. Here, the goal is to get insights into the way in which neural representations shape the final task of phone classification so that it can be used further to explain the loss of intelligibility in disordered speech. In this way, an original framework is proposed, relying firstly on the neural activity and a novel representation per neuron, here considering the phone classification, and, secondly, permitting to identify a set of neurons devoted to the detection of specific phonetic traits on normal speech. Faced to disordered speech, a degradation of that set of neurons is observed, demonstrating a loss of specific phonetic traits in some patients involved, and the potentiality of the proposed approaches to inform about speech alteration.

Exploring audiovisual speech perception in monolingual and bilingual children in Uzbekistan

“…To accomplish this, we calculated the Sørensen-Dice Index (SDI; Sørensen, 1948; Abderrazek et al, 2022), which is a similarity measure that takes into account both the shared elements and the size of the sets being compared to measure similarity between phoneme inventories of several language pairs. The SDI ranges from 0 to 1, where 0 indicates no similarity, and 1 indicates complete similarity.…”

Validation of the Neuro-Concept Detector framework for the characterization of speech disorders: A comparative study including Dysarthria and Dysphonia

“…In this section, we briefly describe the general analytic framework, Neuro-Concept Detector (NCD), proposed in [20]. This framework was designed for the interpretability of the deep rep-resentations of a DNN performing a classification task.…”

“…Conversely, if Sn,T x < −0.25, then the neuron n is considered as a detector of the opposite phonetic feature Tx, noted [-Tx]. Experiments conducted in [20] revealed interesting results. Indeed, it showed that interpretable neurons with phonetic feature encoding properties emerge in the fully connected layers of the CNN.…”

“…From our knowledge, we can mention [18], involving adults suffering from laryngeal cancers and children with cleft lip and palate, or [19] involving dysarthric and dysphonic speakers, children with cleft lip or palate, speakers with pathological speech secondary to hearing impairment, laryngectomized and glossectomized speakers. Recently, we have proposed an original framework, Neuro-Concept Detector (NCD), for the interpretability of Deep Neural Networks (DNNs) [20]. This framework highlights the ability of hidden neurons to detect a specific concept related to the final task.…”