Analyzing the performance of ASR systems

Rodrigues, Ana; Santos, Rita; Abreu, Jorge Ferraz de; Beça, Pedro; Almeida, Pedro; Fernandes, Sílvia

doi:10.1145/3335595.3335635

Cited by 7 publications

(4 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This also underlines that an assessment of ASR performance should not only rely on WER but should be supplemented by some content analysis (see also Proksch, Wratil, and Wäckerle 2019). In line with previous research (Morgan, Wegmann, and Cohen 2013;Rodrigues et al 2019), background noise significantly reduced accuracy and validity (H3 confirmed). When coding the voice-recordings, we also coded what type of background noise appeared and found that background noises are rarely due to social settings (e.g., café, workplace sounds) or outside locations (e.g., traffic noise) but due to, for example, television sounds or white noise.…”

Section: Discussionsupporting

confidence: 79%

“…Voice-recordings can be implemented in web surveys where respondents also answer on PCs, laptops, or tablets. Mobile respondents are more used to voice-recordings than PC or tablet users, which improves ASR performance (Revilla and Couper 2021;Rodrigues et al 2019). Previous research also indicates that ASR performance decreases with increasing distance between the microphone and the speaker (Renals and Swietojanski 2017;Kim et al 2019).…”

Section: Introductionmentioning

confidence: 99%

“…Context factors such as background noise can reduce ASR performance (Morgan, Wegmann, and Cohen 2013;Rodrigues et al 2019).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Keep the noise down: On the performance of automatic speech recognition of voice-recordings in web surveys

Meitinger,

van der Sluis,

Schonlau

2024

Surv Pract

View full text Add to dashboard Cite

Voice-recordings are increasingly implemented in web surveys, but the resulting audio data need to be transcribed before analysis. Since manual coding is too time- and work-intensive, researchers often rely on automatic speech recognition (ASR) systems for the transcription of the voice-recordings. However, ASR tools might create partly incorrect transcriptions and potentially change the content of responses. If the ASR performance (i.e., accuracy and validity) differs by subgroup and contextual factors, a bias is introduced in the analysis of open-ended questions. We assessed the impact of sociodemographic and contextual factors on the accuracy and validity of ASR transcriptions with data from the Longitudinal Internet Studies for the Social Sciences (LISS) panel collected in December 2020. We find that background noise reduces the accuracy and validity of ASR transcriptions. In addition, validity improved when the respondent was alone during the survey. Fortunately, we did not find any evidence of systematic differences across subgroups (age, sex, education), devices or respondent location.

show abstract

Section: Discussionsupporting

confidence: 79%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Keep the noise down: On the performance of automatic speech recognition of voice-recordings in web surveys

Meitinger,

van der Sluis,

Schonlau

2024

Surv Pract

View full text Add to dashboard Cite

show abstract

“…The distance between the speaker and the speech recognition system can impact audio quality and accuracy. Rodrigues et al [57] also demonstrated that the accuracy of speech recognition was decreased with the increasing distance between the user and the microphone. Nematollahi et al [58] suggested that the lesser distance provides better accuracy outcomes for speech recognition.…”

Section: Discussionmentioning

confidence: 97%

The Synergy between a Humanoid Robot and Whisper: Bridging a Gap in Education

Pande,

Mishra

2023

Electronics

View full text Add to dashboard Cite

Students may encounter problems concentrating during a lecture due to various reasons, which can be related to the educator’s accent or the student’s auditory difficulties. This may lead to reduced participation and poor performance in the class. In this paper, we explored whether the incorporation of the humanoid robot Pepper can help in improving the learning experience. Pepper can capture the audio of a person; however, there is no guarantee of accuracy of the recorded audio due to various factors. Therefore, we investigated the limitations of Pepper’s speech recognition system with the aim of observing the effect of distance, age, gender, and the complexity of statements. We conducted an experiment with eight persons including five females and three males who spoke provided statements at different distances. These statements were classified using different statistical scores. Pepper does not have the functionality to transcribe speeches into text. To overcome this problem, we integrated Pepper with a speech-to-text recognition tool, Whisper, which transcribes speech into text that can be displayed on Pepper’s screen using its service. The purpose of the study is to develop a system where the humanoid robot Pepper and the speech-to-text recognition tool Whisper act in synergy to bridge the gap between verbal and visual communication in education. This system could be beneficial for students as they will better understand the content through the visual representation of the teacher’s spoken words regardless of any hearing impairments and accent problems. The methodology involves recording the participant’s speech, followed by its transcription to text by Whisper, and then evaluation of the generated text using various statistical scores. We anticipate that the proposed system will be able to increase the student’s learning experience, engagement, and immersion in a classroom environment.

show abstract