MultiQT: Multimodal learning for real-time question tracking in speech

Abstract

We address a challenging and practical task of labeling questions in speech in real time during telephone calls to emergency medical services in English, which embeds within a broader decision support system for emergency call-takers. We propose a novel multimodal approach to real-time sequence labeling in speech. Our model treats speech and its own textual representation as two separate modalities or views, as it jointly learns from streamed audio and its noisy transcription into text via automatic speech recognition. Our results show significant gains of jointly learning from the two modalities when compared to text or audio only, under adverse noise and limited volume of training data. The results generalize to medical symptoms detection where we observe a similar pattern of improvements with multimodal learning.

When the machine is wrong. Characteristics of true and false predictions of Out-of-Hospital Cardiac arrests in emergency calls using a machine-learning model

“…The model is currently developed to English speaking EMS. 27 With more languages added the machinelearning models sensitivity may improved. Another trait of the false negative calls was calls, where the caller did not have access to the patient.…”

From the ground truth up: doing AI ethics from practice to principles

“…Though the model was partially open-sourced (Havtorn et al 2020;Maaløe et al 2019), our group did not pursue a technical understanding of explaining how the system worked because we were convinced that winning human trust for the AI decisions would originate and remain within the parameters of performance defined as speed and accuracy. Ultimately, the objectivity -the mathematical certaintieswere not just descriptions of functionality but conceptual rigidities that commanded respect by humiliating human subjectivity and uncertainty.…”

On Assessing Trustworthy AI in Healthcare. Machine Learning as a Supportive Tool to Recognize Cardiac Arrest in Emergency Calls

“…The text output of the language model was then fed to a classifier that predicted whether a cardiac arrest was happening or not (Figure 3). The AI system was applied directly on the audio stream where the only processing made was a short-term Fourier transformation (Havtorn et al, 2020), hence no explicit feature selection was made.…”

“…There is no explanation of how the ML makes its predictions. The company that developed the AI system has some of their work in the open domain (Maaløe et al, 2019;Havtorn et al, 2020). However, the exact details on the ML system used for this use case are not publicly available.…”

“…However, the exact details on the ML system used for this use case are not publicly available. The general principles used for this AI system are documented in the study by (Havtorn et al, 2020). The paper describes the AI model implemented for this use case.…”