Multimodal Crowdsourcing for Transcribing Handwritten Documents

Abstract

Abstract-Transcription of handwritten documents is an important research topic for multiple applications, such as document classification or information extraction. In the case of historical documents, their transcription allows to preserve cultural heritage because of the amount of historical data contained in those documents. The transcription process can employ state-ofthe-art handwritten text recognition systems in order to obtain an initial transcription. This transcription is usually not good enough for the quality standards, but that may speed up the final transcription of the expert. In this framework, the use of collaborative transcription applications (crowdsourcing) has risen in the recent years, but these platforms are mainly limited by the use of non-mobile devices. Thus, the recruiting initiatives get reduced to a smaller set of potential volunteers. In this work, an alternative that allows the use of mobile devices is presented. The proposal consists of using speech dictation of handwritten text lines. Then, by using multimodal combination of speech and handwritten text images, a draft transcription can be obtained, presenting more quality than that obtained by only using handwritten text recognition. The speech dictation platform is implemented as a mobile device application, which allows for a wider range of population for recruiting volunteers. A real acquisition on the contents of a Spanish historical handwritten book was obtained with the platform. This data was used to perform experiments on the behaviour of the proposed framework. Some experiments were performed to study how to optimise the collaborators effort in terms of number of collaborations, including how many lines and which lines should be selected for the speech dictation.

Investigating roles of social value orientation and domain knowledge in crowdsourced manuscript transcription

“…Domain knowledge can be defined as the degree of subject familiarity with participants (Ju, ). Previous research indicated that participants with relevant domain knowledge usually provide high quality transcripts (Granell & Martinez‐Hinarejos, ). Thus, we believe that domain knowledge plays an important role in affecting the task completion of crowdsourced manuscript transcription.…”

Multimodality, interactivity, and crowdsourcing for document transcription

“…This system was improved by adding a line selection module that allows us to optimize the collaboration effort (CE) . Then, it was tested in a real scenario where the speech collaborations were acquired from mobile devices …”

“…28 Then, it was tested in a real scenario where the speech collaborations were acquired from mobile devices. 7 In this work, the data fusion and lattice transformation performed by this crowdsourcing framework is studied. Moreover, we experimented with the integration of the crowdsourcing framework and the CATTI system.…”

“…As an alternative to that, volunteers could employ voice as input for transcription, given a speech dictation of the handwritten text contents. This modality is available in nearly all mobile devices and would allow us to obtain a larger number of volunteers . However, voice presents an ambiguity that typed input lacks, since automatic speech recognition (ASR) systems are far from perfect, although their performance has increased substantially over the recent years …”

“…This modality is available in nearly all mobile devices and would allow us to obtain a larger number of volunteers. 7 However, voice presents an ambiguity that typed input lacks, since automatic speech recognition (ASR) 8 systems are far from perfect, although their performance has increased substantially over the recent years. 9 In any case, since paleographers must finally supervise the transcription, the use of voice input could provide a new more accurate transcription (taking as a starting point an HTR transcription) that can reduce substantially the supervision effort of final transcribers.…”