Transcribing and Annotating Speech Corpora for Speech Recognition:  A Three-Step Crowdsourcing Approach with Quality Control

Hämäläinen, Annika; Moreira, Fernando Pinto; Avelar, Jairo; Braga, Daniela; Dias, Miguel Sales

doi:10.1609/hcomp.v1i1.13102

Cited by 2 publications

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

Effective Crowdsourced Generation of Training Data for Chatbots Natural Language Understanding

Bapat

Kucherbaev

Bozzon

2018

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Chatbots are text-based conversational agents. Natural Language Understanding (NLU) models are used to extract meaning and intention from user messages sent to chatbots. The user experience of chatbots largely depends on the performance of the NLU model, which itself largely depends on the initial dataset the model is trained with. The training data should cover the diversity of real user requests the chatbot will receive. Obtaining such data is a challenging task even for big corporations. We introduce a generic approach to generate training data with the help of crowd workers, we discuss the approach workflow and the design of crowdsourcing tasks assuring high quality. We evaluate the approach by running an experiment collecting data for 9 different intents. We use the collected training data to train a natural language understanding model. We analyse the performance of the model under different training set sizes for each intent. We provide recommendations on selecting an optimal confidence threshold for predicting intents, based on the cost model of incorrect and unknown predictions.

show abstract

Effective Crowdsourced Generation of Training Data for Chatbots Natural Language Understanding

Bapat

Kucherbaev

Bozzon

2018

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

Best Practices for Crowdsourcing Dialectal Arabic Speech Transcription

Wray

Mubarak

Ali

2015

Proceedings of the Second Workshop on Arabic Natural Language Processing

View full text Add to dashboard Cite

In this paper, we investigate different approaches in crowdsourcing transcriptions of Dialectal Arabic speech with automatic quality control to ensure good transcription at the source. Since Dialectal Arabic has no standard orthographic representation, it is very challenging to perform quality control. We propose a complete recipe for speech transcription quality control that includes using output of an Automatic Speech Recognition system. We evaluated the quality of the transcribed speech and through this recipe, we achieved a reduction in transcription error of 1.0% compared with 13.2% baseline with no quality control for Egyptian data, and down to 4% compared with 7.8% for the North African dialect.

show abstract

Transcribing and Annotating Speech Corpora for Speech Recognition: A Three-Step Crowdsourcing Approach with Quality Control

Cited by 2 publications

References 1 publication

Effective Crowdsourced Generation of Training Data for Chatbots Natural Language Understanding

Effective Crowdsourced Generation of Training Data for Chatbots Natural Language Understanding

Best Practices for Crowdsourcing Dialectal Arabic Speech Transcription

Contact Info

Product

Resources

About