SMS based interface for FAQ retrieval

Abstract

Short Messaging Service (SMS) is popularly used to provide information access to people on the move. This has resulted in the growth of SMS based Question Answering (QA) services. However automatically handling SMS questions poses significant challenges due to the inherent noise in SMS questions. In this work we present an automatic FAQ-based question answering system for SMS users. We handle the noise in a SMS query by formulating the query similarity over FAQ questions as a combinatorial search problem. The search space consists of combinations of all possible dictionary variations of tokens in the noisy query. We present an efficient search algorithm that does not require any training data or SMS normalization and can handle semantic variations in question formulation. We demonstrate the effectiveness of our approach on two reallife datasets.

Variant search and syntactic tree similarity based approach to retrieve matching questions for SMS queries

“…Lucene [1] uses fuzzy searcher based on the Levenshtein distance. However, as reported in [6] with higher value of similarity parameter in Lucene's fuzzy match, the performance of information retrieval actually degrades. To the best of our knowledge the method proposed by [6] is the latest work in the development of SMS based automatic question answering in which they propose the following similarity measure between a SMS term (s i) and a term (t) in the domain dictionary:…”

“…To measure the effectiveness of our system, we tested our system on the SMS query set collected from IBM, India Research Lab. The query set was obtained from the authors of [6] who generated it by asking human evaluators to choose questions randomly from the FAQ dataset. The evaluators typed the selected questions as SMS queries on a mobile keypad interface.…”

“…Govind et.al. [6] propose an unsupervised approach for handling noisy lexical and semantic variations in SMS queries. Previous approaches rely on aligned corpus of SMS and conventional language for training which is difficult to build and requires considerable human effort.…”

Rule Based Automatic Generation of Query Terms for SMS Based Retrieval Systems

“…An aligned SMS corpus and conventional language is required for training by the techniques employed by them. As reported by [4], Acharya et al, 2008 worked towards mapping non-standard words to their corresponding conformist recurrent form through an unsupervised technique. The algorithm proposed by Govind Kothari et al [4] takes care of the noise in a SMS query by formulating query similarity over FAQ questions along with handling semantic variations in question formulation.…”

“…As reported by [4], Acharya et al, 2008 worked towards mapping non-standard words to their corresponding conformist recurrent form through an unsupervised technique. The algorithm proposed by Govind Kothari et al [4] takes care of the noise in a SMS query by formulating query similarity over FAQ questions along with handling semantic variations in question formulation. As his approach considers it as a combinatorial search problem, therefore, the search space consists of combinations of all possible dictionary variations of tokens in the SMS query.…”