Automatic assessment of voice quality in the context of multiple annotations

Abstract

Approaches to evaluate voice quality include perceptual analysis, and acoustical analysis. Perceptual analysis is subjective and depends mostly on the ability of a specialist to assess a pathology, whereas acoustical analysis is objective, but highly relies on the quality of the so called annotations that the specialist assigns to the voice signal. The quality of the annotations for acoustical analysis depends heavily on the expertise and knowledge of the specialist. We face a scenario where we have annotations performed by several specialists with different levels of expertise and knowledge. Traditional pattern recognition methods employed in acoustical analysis are no longer applicable, since these methods are designed for scenarios where a "ground-truth" label is assigned by the specialist. In this paper, we apply recent developments in machine learning for taking into account multiple annotators for acoustical analysis of voice signals. For the classification step we compare two techniques, one of them based on Gaussian Processes for regression with multiple annotators, and the other is a multi-class Logistic Regression model that measures the annotator performance in terms of sensitivity and specificity. The performance of classifiers is assessed in terms of Cohen's Kappa index. Results show that the multi-annotator classification schemes have better performance when compared to techniques based on a traditional classifier where the true label is estimated from the multiple annotations available using majority voting.

Artificial intelligence for ultrasound scanning in regional anaesthesia: a scoping review of the evidence from multiple disciplines

“…There is a notable change in the AI techniques used around 2016–17. Early approaches, most published before 2016–17, typically involved steps of feature extraction from the ultrasound image, followed by classification of these features, for both PNB 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 and CNB. 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 Around this point, there has been an increase in published data where the dominant technique was deep learning for both PNB 9 , 11 , 12 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 , 73 and CNB, 7 , 74 ,…”

“…In the available sources, a variety of human image interpretation was used as the ground truth. This included a range in the number of humans used in the ground truth from one, 7 , 23 , 24 , 25 , 31 , 32 , 34 , 35 , 36 , 38 , 46 , 49 , 58 , 60 , 67 , 72 , 77 two, 8 , 9 , 19 , 21 , 26 , 30 , 33 , 44 , 61 three, 51 , 52 , 53 , 63 , 65 and up to 15. 48 Some sources cited ‘multiple’ humans, used a plural term (e.g.…”

“…Sources described their ground truth human(s) as a volunteer, 64 doctor/physician, 48 , 52 , 63 , 65 sonographer, 7 , 30 , 31 , 32 , 33 , 34 , 35 , 60 , 75 , 77 clinical professional, 27 , 38 or (regional) anaesthetist. 8 , 17 , 18 , 19 , 20 , 21 , 23 , 25 , 26 , 36 , 44 , 45 , 46 , 49 , 51 , 53 , 54 , 58 , 61 , 62 , 67 , 69 The level of expertise of those providing the ground truth was described as trained, 64 experienced, 21 , 30 , 31 , 32 , 33 , 34 , 46 , 48 , 52 , 65 expert, 7 , 8 , 17 , 18 , 19 , 20 , 25 , 26 , 27 , 33 , …”

Machine learning from crowds: A systematic review of its applications

“…Other interesting applications in this area include: estimation of respiratory rate from the gene normalization (Lu et al, ); estimation of fetal heart rate, interbeat intervals and fetal QT intervals with noninvasive electrocardiograms (ECG) (Silva et al, ); protein folding (Peng et al, ); ECG signal classification (Zhu et al, ; Zhu, Dunkley, et al, ); photoplethysmograms (Zhu, Pimentel, et al, ); sleep spindle detection (Tan et al, ); assessment of voice pathologies (González et al, ) and learning for ICD‐11 sanctioning rules (Lou et al, ); labeling respiratory patterns (Robles‐Rubio et al, ); analysis of respiratory data (Zhu et al, ); and learning rules for disease‐remedy associations (Someswar & Bhattacharya, ).…”

“…Related to this, in Shashidhar et al (), the authors also express the need to develope real‐time crowdsourcing algorithms. There is also interest in using crowdsourcing with multilabel (González et al, ; Mavandadi et al, ), multi‐instance (Tu et al, ), and imbalanced problems (Hernández‐González et al, ).…”