Improved open-vocabulary spoken content retrieval with word and subword lattices using acoustic feature similarity

Lee, Hung-yi; Chou, Po-Wei; Lee, Lin-Shan

doi:10.1016/j.csl.2013.12.003

Cited by 9 publications

(15 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It also yielded 13% relative improvement for a set of OOV queries on audio recordings of McGill course lectures [193] with several speakers [185], and 6.1% relative improvements on broadcast news with many speakers [184]. The graph-based approach with random walk was also shown to outperform the exemplar-based approach with examples from PRF [181]. This is because the exemplarbased approach only considers those information for objects most confident to be relevant or irrelevant, whereas the graphbased approach globally considers all the objects retrieved in the first pass.…”

Section: E Graph-based Approachmentioning

confidence: 94%

“…Another way to exploit the graph structure is using the random walk [181]- [183], [185], which does not use any labelled data. The basic idea is that the hypothesized regions (nodes) strongly connected to many other hypothesized regions (nodes) with higher/lower confidence scores on the graph should have higher/lower scores.…”

Section: E Graph-based Approachmentioning

confidence: 99%

“…One way is to represent the acoustic vector sequences by fixed length feature vectors as in Fig 5 [180], and then compute the similarity between the two fixed length feature vectors. Another way is to use dynamic time warping (DTW) to evaluate the similarity between two acoustic vector sequences with different lengths [181]- [183]. DTW will be further discussed later on in Section V. This approach was shown to yield significant improvements on both course lectures [181] and broadcast news [184] 9 .…”

Section: Query-specific Rescoring Based On Pseudo-relevance Feedbamentioning

confidence: 99%

“…Another way is to use dynamic time warping (DTW) to evaluate the similarity between two acoustic vector sequences with different lengths [181]- [183]. DTW will be further discussed later on in Section V. This approach was shown to yield significant improvements on both course lectures [181] and broadcast news [184] 9 . This approach was also shown to offer improvement additive to the retrieval-oriented acoustic modeling under relevance feedback scenario in Section III-B3 [129].…”

Section: Query-specific Rescoring Based On Pseudo-relevance Feedbamentioning

confidence: 99%

See 3 more Smart Citations

Spoken Content Retrieval—Beyond Cascading Speech Recognition with Text Retrieval

Lee

Glass

Lee

et al. 2015

IEEE/ACM Trans. Audio Speech Lang. Process.

Self Cite

100

View full text Add to dashboard Cite

Spoken content retrieval refers to directly indexing and retrieving spoken content based on the audio rather than text descriptions. This potentially eliminates the requirement of producing text descriptions for multimedia content for indexing and retrieval purposes, and is able to precisely locate the exact time the desired information appears in the multimedia. Spoken content retrieval has been very successfully achieved with the basic approach of cascading automatic speech recognition (ASR) with text information retrieval: after the spoken content is transcribed into text or lattice format, a text retrieval engine searches over the ASR output to find desired information. This framework works well when the ASR accuracy is relatively high, but becomes less adequate when more challenging real-world scenarios are considered, since retrieval performance depends heavily on ASR accuracy. This challenge leads to the emergence of another approach to spoken content retrieval: to go beyond the basic framework of cascading ASR with text retrieval in order to have retrieval performances that are less dependent on ASR accuracy. This overview article is intended to provide a thorough overview of the concepts, principles, approaches, and achievements of major technical contributions along this line of investigation. This includes five major directions: 1) Modified ASR for Retrieval Purposes: cascading ASR with text retrieval, but the ASR is modified or optimized for spoken content retrieval purposes; 2) Exploiting the Information not present in ASR outputs: to try to utilize the information in speech signals inevitably lost when transcribed into phonemes and words; 3) Directly Matching at the Acoustic Level without ASR: for spoken queries, the signals can be directly matched at the acoustic level, rather than at the phoneme or word levels, bypassing all ASR issues; 4) Semantic Retrieval of Spoken Content: trying to retrieve spoken content that is semantically related to the query, but not necessarily including the query terms themselves; 5) Interactive Retrieval and Efficient Presentation of the Retrieved Objects: with efficient presentation of the retrieved objects, an interactive retrieval process incorporating user actions may produce better retrieval results and user experiences.Index Terms-Spoken content retrieval, spoken term detection, query by example, semantic retrieval, joint optimization, pseudo-relevance feedback, graph-based random walk, unsupervised acoustic pattern discovery, query expansion, interactive retrieval, summarization, key term extraction.

show abstract

Section: E Graph-based Approachmentioning

confidence: 94%

Section: E Graph-based Approachmentioning

confidence: 99%

Section: Query-specific Rescoring Based On Pseudo-relevance Feedbamentioning

confidence: 99%

Section: Query-specific Rescoring Based On Pseudo-relevance Feedbamentioning

confidence: 99%

See 2 more Smart Citations

Spoken Content Retrieval—Beyond Cascading Speech Recognition with Text Retrieval

Lee

Glass

Lee

et al. 2015

IEEE/ACM Trans. Audio Speech Lang. Process.

Self Cite

100

View full text Add to dashboard Cite

show abstract

“…where C(d) is the ASR score of d, δ is a interpolation factor tuned on the development data. This method is similar to the Pseudo Relevance Feedback (PRF) [26]. Note that, the PRF assumes that top detections (e.g.…”

Section: Rescoring Oov Detections By Simple Interpolationmentioning

confidence: 99%

Rescoring Hypothesized Detections of Out-of-Vocabulary Keywords Using Subword Samples

Pham

Xiao

et al. 2016

Interspeech 2016

View full text Add to dashboard Cite

Rescoring hypothesized detections, using keyword's audio samples extracted from training data, is an effective way to improve the performance of a Keyword Search (KWS) system. Unfortunately such rescoring framework cannot be applied directly to Out-of-Vocabulary (OOV) keywords since there is no sample in the training data. To address this limitation, we propose two techniques for OOV keywords in this work. The first technique generates samples for an OOV keyword by concatenating samples of its constituent subwords. The second technique splits hypothesized detections into segments, then estimates the acoustic similarities between detections and subword's samples according to the similarities between segments and these samples. The similarity scores from these two techniques are used to rescore and re-rank the list of detections returned by the automatic speech recognition (ASR) systems. The experiments show that incorporating the proposed similarity scores results in a better separation between the correct and false alarm detections than using the ASR scores alone. Furthermore, experimental results on the NIST OpenKWS15 Evaluation show that rescoring with the proposed similarity scores significantly outperforms the raw ASR scores, and other methods that do not use the similarity scores, in both Maximum Term Weighted Value (MTWV) and Mean Average Precision (MAP) metrics.

show abstract

Editorial for the special issue on spoken content retrieval

Saraçlar

Chelba

Ramabhadran³

2014

Computer Speech & Language

View full text Add to dashboard Cite

Improved open-vocabulary spoken content retrieval with word and subword lattices using acoustic feature similarity

Cited by 9 publications

References 57 publications

Spoken Content Retrieval—Beyond Cascading Speech Recognition with Text Retrieval

Spoken Content Retrieval—Beyond Cascading Speech Recognition with Text Retrieval

Rescoring Hypothesized Detections of Out-of-Vocabulary Keywords Using Subword Samples

Editorial for the special issue on spoken content retrieval

Contact Info

Product

Resources

About