Comparison of ALBAYZIN query-by-example spoken term detection 2012 and 2014 evaluations

Abstract

Query-by-example spoken term detection (QbE STD) aims at retrieving data from a speech repository given an acoustic query containing the term of interest as input. Nowadays, it is receiving much interest due to the large volume of multimedia information. This paper presents the systems submitted to the ALBAYZIN QbE STD 2014 evaluation held as a part of the ALBAYZIN 2014 Evaluation campaign within the context of the IberSPEECH 2014 conference. This is the second QbE STD evaluation in Spanish, which allows us to evaluate the progress in this technology for this language. The evaluation consists in retrieving the speech files that contain the input queries, indicating the start and end times where the input queries were found, along with a score value that reflects the confidence given to the detection of the query. Evaluation is conducted on a Spanish spontaneous speech database containing a set of talks from workshops, which amount to about 7 h of speech. We present the database, the evaluation metric, the systems submitted to the evaluation, the results, and compare this second evaluation with the first ALBAYZIN QbE STD evaluation held in 2012. Four different research groups took part in the evaluations held in 2012 and 2014. In 2014, new multi-word and foreign queries were added to the single-word and in-language queries used in 2012. Systems submitted to the second evaluation are hybrid systems which integrate letter transcription-and template matching-based systems. Despite the significant improvement obtained by the systems submitted to this second evaluation compared to those of the first evaluation, results still show the difficulty of this task and indicate that there is still room for improvement.

ALBAYZIN 2016 spoken term detection evaluation: an international open competitive evaluation in Spanish

“…These evaluation campaigns provide an objective mechanism to compare different systems and are a powerful way to promote research on different speech technologies [56][57][58][59][60][61][62][63].…”

ALBAYZIN 2018 spoken term detection evaluation: a multi-domain international evaluation in Spanish

“…ALBAYZIN evaluation campaigns comprise an internationally open set of evaluations supported by the Spanish Thematic Network on Speech Technologies (RTTH 1 ) and the ISCA Special Interest Group on Iberian Languages (SIG-IL 2 ), which have been held biennially since 2006. These evaluation campaigns provide an objective mechanism to compare different systems and are a powerful way to promote research on different speech technologies [77][78][79][80][81][82][83][84][85][86].…”

ALBAYZIN Query-by-example Spoken Term Detection 2016 evaluation

“…These campaigns are internationally open sets of evaluations supported by the Spanish Network of Speech Technologies (RTTH) 1 and the ISCA Special Interest Group on Iberian Languages (SIG-IL) 2 , which have been held every 2 years since 2006. These evaluation campaigns provide an objective mechanism for the comparison of different systems and the promotion of research into different speech technologies such as audio segmentation [51], speaker diarization [52], language recognition [53], spoken term detection [54], query-by-example spoken term detection [55,56], and speech synthesis [57].…”