Information Retrieval Metrics for Speech Based Systems: A Systematic Review

Abstract

Information Retrieval (IR) allows the identification of relevant information from connected repositories, however their performance have been of research interest leading to investigations in the modalities by which the accuracy of the retrievals are evaluated. Metrics such as Precision, Recall, F-score among others are used to evaluate an IR system. IR use same form of evaluation for both speech and text based system while failing to realize the difference that could have occurred in the process of transcription, especially in the voice to text search, which is the most common speech based search paradigm. This is forming a new set of concerns. This research aim to review and identify the strengths and weaknesses of existing metrics for measuring the performances of speech based. A total of 179 articles were retrieved using Google Scholar repository and were manually examined. Only 25 articles were selected for analysis in this study after applying our predefined inclusion and exclusion criteria. Result shows that Mean Average Precision is the most frequently used metric for speech based IR system with result range from 0.4191 to 0.620. Also transcription error of spoken query or spoken document has a near linear relationship to IR performance. This systematic review serves as a bibliography of speech based IR systems and can be used by those new to the field of IR.