Is automatic speech-to-text transcription ready for use in psychological experiments?

Abstract

Verbal responses are a convenient and naturalistic way for participants to provide data in psychological experiments (Salzinger, 1959). However, audio recordings of verbal responses typically require additional processing, such as transcribing the recordings into text, as compared with other behavioral response modalities (e.g. typed responses, button presses, etc.). Further, the transcription process is often tedious and time-intensive, requiring human listeners to manually examine each moment of recorded speech. Here we evaluate the performance of a state-of-the-art speech recognition algorithm (Halpern et al., 2016) in transcribing audio data into text during a list-learning experiment. We compare transcripts made by human annotators to the computer-generated transcripts. Both sets of transcripts matched to a high degree and exhibited similar statistical properties, in terms of the participants' recall performance and recall dynamics that the transcripts captured. This proof-of-concept study suggests that speech-to-text engines could provide a cheap, reliable, and rapid means of automatically transcribing speech data in psychological experiments. Further, our findings open the door for verbal response experiments that scale to thousands of participants (e.g. administered online), as well as a new generation of experiments that decode speech on-the-fly and adapt experimental parameters based on participants' prior responses.

Vocabulary Size Influences Spontaneous Speech in Native Language Users: Validating the Use of Automatic Speech Recognition in Individual Differences Research

“…Instead, we intend to provide a proof-of-concept that an ASR can be used to analyze certain aspects of spontaneous speech, allowing for large-scale use of natural speech for research ends. A similar approach has recently been taken by Ziman et al (2018), who showed that an ASR can be used reliably to transcribe speech data from psychological experiments, in their case a verbal recall memory test. In their study, Ziman and colleagues provided the speech context to their speech-to-text engine.…”

“…The strength of the correlations might be considered an index of how well a given measure, in the context of spontaneous speech elicitation, is suited to be transcribed by an ASR, or whether it may require manual coding. (For a similar correlational approach to evaluate transcription accuracy, see Ziman et al, 2018.) As we planned to carry out correlations for many measures of interest, we applied a Bonferroni correction (four measures and three questions resulted in a corrected alpha level of 0.05/12 = 0.004).…”

Social Reminiscence in Older Adults’ Everyday Conversations: Automated Detection Using Natural Language Processing and Machine Learning

“…The use of different NLP features, classifiers, and learning strategies discussed in this study seems promising to develop a system for the real-time detection of reminiscence in everyday conversations in German of older adults. Such a system could leverage audio-to-text software [ 78 ] of advanced methods from automated coding [ 24 ] to automate the transcription of conversations before NLP preprocessing and the computation of machine learning predictions.…”

From The Daily Show to Last Week Tonight: A Quantitative Analysis of Discursive Integration in Satirical Television News

“…Transcripts of the satirical news shows and the liberal and conservative news shows were collected by means of the command-line program youtube-dl (available at: http://ytdl-org.github.io/youtube-dl/), which we used to download automatic captions from YouTube. We used YouTube because previous research has found such automatic speech-to-text transcriptions to be accurate (Ziman et al 2018). In some respects, they may even be more reliable than the original US television subtitles because real-time subtitles can contain typos and are subject to strict character and time restrictions (Szarkowska, Cintas, and Gerber-Morón in press).…”