Real-time synchronization of live speech with its transcription

Abstract

Most of the researches in synchronization of audio and text have been focusing on the synchronization at the level of utterance. However, to generate audio books in unstructured language like Thai from live speech, a finer level of synchronization is necessary. We propose an algorithm to synchronize live speech with its corresponding transcription in real time at syllabic unit. The proposed algorithm employs the syllable endpoint detection method and the syllable landmark detection method with bandlimited intensity as features. The experiment was conducted with LOTUS datasets and the results were compared with baseline ASR-based syllable detection. We evaluated our algorithm by measuring its error through error aberration, which is the difference of the actual number of syllables and the detected syllables for each phrase, and found average total error aberration of the proposed algorithm to outperform that of the baseline. The average total error aberrations are 11.54 and 34.21 for the proposed method and the baseline respectively. We also found the reference deviation from our method to be better than that of the baseline as well.

An Automatic Real-time Synchronization of Live speech with Its Transcription Approach

“…The results of the real-time syllable detection procedure. Approaches Acoustic Feature Sets Accuracy Our Previous work [14] -82.02%…”

“…For Baseline 1, we considered the work from [14] and found its weakness in landmark detection. In that work, the audio stream was continuously passed into the Real-Time Syllable Endpoint Detection module that extract the feature representation of each speech frame to determine whether the current speech frame was the endpoint, which means that the current syllable transcription has already been spoken.…”

“…The energy threshold and the dip threshold of the convex hull algorithm are set differently for each approach according to the training data. The previous work [14] set the parameters to 60 dB and 10 dB respectively. Our proposed SVM-based approach with the first acoustic feature set the parameters to 56 dB and 6 dB respectively, whereas the SVM-based approach with the second acoustic feature set the parameters to 60 dB and 8 dB respectively.…”

“…4, the starting point of syllable is at the point A, at which the intensity of a speech frame is greater than the threshold. Based upon our preliminary experiment on the training dataset, we define the threshold at 56 dB, since it provides the best results as reported in our previous work [14]. As we mentioned, the endpoint should be at the point B, at which the intensity of a speech frame is less than the threshold.…”

“…Several studies for automatic synchronization of transcription and speech at the utterance level mostly concern in the non-real-time situation. In our previous work [14], an algorithm for the automatic synchronization of live speech and its transcriptions in the real-time situation at the syllabic level is introduced. Although our previous work provides reasonable results, the results still are some drawbacks of our previous approach because it only uses audio data to calculate.…”