Spoken Content Retrieval—Beyond Cascading Speech Recognition with Text Retrieval

Abstract

Spoken content retrieval refers to directly indexing and retrieving spoken content based on the audio rather than text descriptions. This potentially eliminates the requirement of producing text descriptions for multimedia content for indexing and retrieval purposes, and is able to precisely locate the exact time the desired information appears in the multimedia. Spoken content retrieval has been very successfully achieved with the basic approach of cascading automatic speech recognition (ASR) with text information retrieval: after the spoken content is transcribed into text or lattice format, a text retrieval engine searches over the ASR output to find desired information. This framework works well when the ASR accuracy is relatively high, but becomes less adequate when more challenging real-world scenarios are considered, since retrieval performance depends heavily on ASR accuracy. This challenge leads to the emergence of another approach to spoken content retrieval: to go beyond the basic framework of cascading ASR with text retrieval in order to have retrieval performances that are less dependent on ASR accuracy. This overview article is intended to provide a thorough overview of the concepts, principles, approaches, and achievements of major technical contributions along this line of investigation. This includes five major directions: 1) Modified ASR for Retrieval Purposes: cascading ASR with text retrieval, but the ASR is modified or optimized for spoken content retrieval purposes; 2) Exploiting the Information not present in ASR outputs: to try to utilize the information in speech signals inevitably lost when transcribed into phonemes and words; 3) Directly Matching at the Acoustic Level without ASR: for spoken queries, the signals can be directly matched at the acoustic level, rather than at the phoneme or word levels, bypassing all ASR issues; 4) Semantic Retrieval of Spoken Content: trying to retrieve spoken content that is semantically related to the query, but not necessarily including the query terms themselves; 5) Interactive Retrieval and Efficient Presentation of the Retrieved Objects: with efficient presentation of the retrieved objects, an interactive retrieval process incorporating user actions may produce better retrieval results and user experiences.Index Terms-Spoken content retrieval, spoken term detection, query by example, semantic retrieval, joint optimization, pseudo-relevance feedback, graph-based random walk, unsupervised acoustic pattern discovery, query expansion, interactive retrieval, summarization, key term extraction.

VTLN-warped Gaussian posteriorgram for QbE-STD

“…All the queries are distributed across all the speakers such that at least one speaker contains at least one query. The performance of QbE-STD is measured in terms of precision@N (p@N) and Mean Average Precision (MAP) [2]. The value of N varies according to the query (from 7 to 20).…”

“…QbE-STD directly exploits the acoustic-level information for matching between spoken documents and a spoken query without transcribing them into phonemes or words. QbE-STD is important for low-resourced languages and under non-mainstream conditions and hence, it was also called an unsupervised STD [1], [2]. As a part of the MediaEval campaign, the Spoken Web Search (SWS) was started in 2011 [3].…”

Usage of acoustic cues in spoken term detection keyword spotting for zero low resource languages

“…These systems can be broadly classified into following categories [2]. Cascaded ASR with text information retrieval: The spoken content is converted into word or sub word sequences or lattices using ASR and then text retrieval techniques are applied.…”

An Intelligent System for Spoken Term Detection That Uses Belief Combination

“…Nowadays, it is receiving much importance due to the large volume of multimedia information. Research and technology improvements in automated speech recognition successfully achieved the information retrieval by using the transcribed textual form of the spoken contents [1]. Similarly, due to the exponential growth of internet and multimedia contents, the STD methods have been achieving much popularity.…”