Automatic Screening Of Children With Speech Sound Disorders Using Paralinguistic Features

Shahin, Mostafa; Ahmed, Beena; Smith, Daniel; Duenser, Andreas; Epps, Julien

doi:10.1109/mlsp.2019.8918725

Cited by 5 publications

(7 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Long-term average spectra for each recording were inspected for possible noise artefacts and further cleaned if any were found (Olsen, 2018). Following Shahin et al (2019), we extracted the standardized voice feature set eGeMAPS with the open-source software OpenSmile (Eyben, 2015, p. 201; Eyben et al, 2010, 2016). The feature set contains 87 features, which are described in Appendix A2, Table S2.…”

Section: Methodsmentioning

confidence: 99%

“…We focused on relatively simple models, as they tend to be less prone to overfitting, therefore, if we were to find generalizability concerns (i.e., inability to accurately identify autistic participants in new samples), such concerns would be even more relevant for models like neural networks, which are more complex and more likely to overfit. For our methodological model we were mainly inspired by Shahin et al (2019), a study based on SVM and rigorous cross-validation (CV), reaching high performance in predicting autism from voice. The authors reported an accuracy of 0.88, that is, 88% of the samples were correctly classified, and an F1 score of 0.90.…”

Section: Methodsmentioning

confidence: 99%

“…This study sets the methodological choices from Shahin et al (2019) in a highly conservative and fully reported pipeline, which relies on cross-validated training procedures and held-out testing sets, that is, it ensures the model is trained (fitted) on a subset of the data (training dataset), but its performance is assessed on a subset of the data on which it has not been fitted (held-out dataset). Figure 1 provides an overview of the pipeline, which is discussed in detail below.…”

Section: Pipelinementioning

confidence: 99%

See 2 more Smart Citations

Vocal markers of autism: assessing the generalizability of machine learning models

Rybner

Jessen

Mortensen

et al. 2021

Preprint

View full text Add to dashboard Cite

Machine learning (ML) approaches show increasing promise in their ability to identify vocal markers of autism. Nonetheless, it is unclear to what extent such markers generalize to new speech samples collected e.g., using a different speech task or in a different language. In this paper, we systematically assess the generalizability of ML findings across a variety of contexts. We train promising published ML models of vocal markers of autism on novel cross-linguistic datasets following a rigorous pipeline to minimize overfitting, including cross-validated training and ensemble models. We test the generalizability of the models by testing them on i) different participants from the same study, performing the same task; ii) the same participants, performing a different (but similar) task; iii) a different study with participants speaking a different language, performing the same type of task.While model performance is similar to previously published findings when trained and tested on data from the same study (out-of-sample performance), there is considerable variance between studies. Crucially, the models do not generalize well to different, though similar, tasks and not at all to new languages. The ML pipeline is openly shared.Generalizability of ML models of vocal markers - and more generally biobehavioral markers - of autism is an issue. We outline three recommendations for strategies researchers could take to be more explicit about generalizability and improve it in future studies.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Pipelinementioning

confidence: 99%

See 1 more Smart Citation

Vocal markers of autism: assessing the generalizability of machine learning models

Rybner

Jessen

Mortensen

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…This study sets the methodological choices from Shahin et al (2019) in a highly conservative and fully reported pipeline, which relies on cross-validated training procedures and held-out testing sets, that is, it ensures the model is trained (fitted) on one subset of the data (the training dataset), while its performance is assessed on a different subset of the data (the held-out dataset). Figure 1 provides an overview of the pipeline, which is discussed in detail below.…”

Section: Pipelinementioning

confidence: 99%

“…Long-term average spectra for each recording were inspected for possible noise artifacts and further cleaned if any were found (Olsen, 2018). Following Shahin et al (2019), we extracted the standardized voice feature set eGeMAPS with the open-source software OpenSmile (Eyben, 2015, p. 201;Eyben et al, 2010Eyben et al, , 2016. The feature set contains 87 features, which are described in Appendix A2, Table S2.…”

Section: Preprocessingmentioning

confidence: 99%

Vocal markers of autism: Assessing the generalizability of machine learning models

et al. 2022

View full text Add to dashboard Cite

Machine learning (ML) approaches show increasing promise in their ability to identify vocal markers of autism. Nonetheless, it is unclear to what extent such markers generalize to new speech samples collected, for example, using a different speech task or in a different language. In this paper, we systematically assess the generalizability of ML findings across a variety of contexts. We train promising published ML models of vocal markers of autism on novel cross‐linguistic datasets following a rigorous pipeline to minimize overfitting, including cross‐validated training and ensemble models. We test the generalizability of the models by testing them on (i) different participants from the same study, performing the same task; (ii) the same participants, performing a different (but similar) task; (iii) a different study with participants speaking a different language, performing the same type of task. While model performance is similar to previously published findings when trained and tested on data from the same study (out‐of‐sample performance), there is considerable variance between studies. Crucially, the models do not generalize well to different, though similar, tasks and not at all to new languages. The ML pipeline is openly shared. Generalizability of ML models of vocal markers of autism is an issue. We outline three recommendations for strategies researchers could take to be more explicit about generalizability and improve it in future studies. Lay Summary Machine learning approaches promise to be able to identify autism from voice only. These models underestimate how diverse the contexts in which we speak are, how diverse the languages used are and how diverse autistic voices are. Machine learning approaches need to be more careful in defining their limits and generalizability.

show abstract

Predicting developmental language disorders using artificial intelligence and a speech data analysis tool

Beccaluva

Catania

Arosio

et al. 2023

Human–Computer Interaction

View full text Add to dashboard Cite

Automatic Screening Of Children With Speech Sound Disorders Using Paralinguistic Features

Cited by 5 publications

References 15 publications

Vocal markers of autism: assessing the generalizability of machine learning models

Vocal markers of autism: assessing the generalizability of machine learning models

Vocal markers of autism: Assessing the generalizability of machine learning models

Predicting developmental language disorders using artificial intelligence and a speech data analysis tool

Contact Info

Product

Resources

About