Alzheimer’s Dementia Speech (Audio vs. Text): Multi-Modal Machine Learning at High vs. Low Resolution

Abstract

Automated techniques to detect Alzheimer’s Dementia through the use of audio recordings of spontaneous speech are now available with varying degrees of reliability. Here, we present a systematic comparison across different modalities, granularities and machine learning models to guide in choosing the most effective tools. Specifically, we present a multi-modal approach (audio and text) for the automatic detection of Alzheimer’s Dementia from recordings of spontaneous speech. Sixteen features, including four feature extraction methods (Energy–Time plots, Keg of Text Analytics, Keg of Text Analytics-Extended and Speech to Silence ratio) not previously applied in this context were tested to determine their relative performance. These features encompass two modalities (audio vs. text) at two resolution scales (frame-level vs. file-level). We compared the accuracy resulting from these features and found that text-based classification outperformed audio-based classification with the best performance attaining 88.7%, surpassing other reports to-date relying on the same dataset. For text-based classification in particular, the best file-level feature performed 9.8% better than the frame-level feature. However, when comparing audio-based classification, the best frame-level feature performed 1.4% better than the best file-level feature. This multi-modal multi-model comparison at high- and low-resolution offers insights into which approach is most efficacious, depending on the sampling context. Such a comparison of the accuracy of Alzheimer’s Dementia classification using both frame-level and file-level granularities on audio and text modalities of different machine learning models on the same dataset has not been previously addressed. We also demonstrate that the subject’s speech captured in short time frames and their dynamics may contain enough inherent information to indicate the presence of dementia. Overall, such a systematic analysis facilitates the identification of Alzheimer’s Dementia quickly and non-invasively, potentially leading to more timely interventions and improved patient outcomes.

Performance Assessment of ChatGPT versus Bard in Detecting Alzheimer’s Dementia

“…To facilitate textual input, the speech audio was transcribed into text using the Otter.ai platform [41]. Given the zero-shot nature of our approach, we focused on the 71 recordings (36 CN, 35 AD) constituting the testing set of ADReSSo Challenge [39], aligning with our prior work [10] for comparison and generalizability assessment.…”

“…Among AD subjects, GPT-3.5 demonstrates a broad range of MMSE scores (8-30) for correctly classified AD subjects yet predicts "Unsure" even for subjects with lower scores (5)(6)(7)(8)(9)(10)(11)(12)(13)(14)(15)(16). Additionally, it occasionally misclassifies AD subjects as CN for slightly higher MMSE scores (12)(13)(14)(15)(16)(17)(18)(19)(20)(21)(22)(23)(24)(25), which may be deemed reasonable.…”

“…Speech characteristics, including fluency, vocabulary usage, and vocal quality, are demonstrably affected by the underlying cognitive decline. By analysing acoustic features defining such speech patterns, researchers are exploring the possibility of developing a non-invasive, objective, and cost-effective method for predicting cognitive decline [10][11][12][13][14][15][16][17]. Furthermore, with the recent advancements in natural language processing (NLP), researchers can now transcribe speech to text and analyze the resulting text to predict cognitive decline [18][19][20][21][22].…”

“…In our recent work [10], we adopted a multi-modal approach that integrated audio-and text-based techniques to automatically detect AD from audio recordings of spontaneous speech. Notably, our text-based classification method exhibited superior performance (88.7% accuracy) compared to our audio-based classification (78.9% accuracy).…”

“…Motivated by this, we investigate here the potential viability of LLM chatbots-in their 'as-is' form (i.e., utilizing zero-shot learning in which the model is expected to make predictions about classes it has not been explicitly trained on [30,31])-as a possible tool to detect AD using only spontaneous speech. While previous studies using audio, text, or their fusion with model embeddings as features achieved promising AD/CN classification accuracies ranging from 78 to 88% [10][11][12]24,28,29,[32][33][34], they all relied on supervised learning procedures with labelled training data. This limits their generalizability to real-world settings lacking readily available labels.…”