A Diary Study-Based Evaluation Framework for Mobile Information Retrieval

Abstract

In this poster, we propose an evaluation framework that investigates the integration of the user context (interests, location and time) into the evaluation process of mobile IR. Our approach is based on a diary study where users are asked to log their queries annotated by their location and time. Users' interests are explicitly acquired or implicitly learned based on users' relevance judgments for the retrieved documents answering their queries. We propose two evaluation protocols namely training/test in chronological order and k-fold cross validation. We exploit this framework in order to evaluate the performance of our context-based personalized mobile search approach. Experimental results show the stability performance of our approach according to the proposed evaluation protocols and demonstrate the viability of the diary approach as a means to capture context in evaluation.

Evaluation in Contextual Information Retrieval

“…This category of evaluation settings includes user studies [94,114,115,124,139,163] and diary studies [11,23,95]. According to the context strati cation model of Ingwersen and Jarvelin [63] shown in Figure 1, user-based evaluation studies focus on dimension 4 without considering interaction/dynamic context.…”

“…In the contextual IR area, few evaluation studies are based on typical diary studies. Beyond analyzing mobile information needs [21,24], diary studies were undertaken [11] to measure retrieval accuracy within mobile environments where location, time, and user interests are the key context attributes. In Reference [21], a diary study was conducted where the data collection is mainly a lifelog data that includes computer activities, mobile phone activities, photos, geo-location, Bluetooth, biometrics, and tweets.…”

“…A review of the -Context: participants are asked to label their information needs with a narrative description, the explicit location, and time they issued their queries. The participants' interests are manually speci ed by participants themselves or automatically learned from their manual relevance assessments of the top returned documents in response to their past queries [11,21]. -Topics and tasks: free topics expressed by the participants in situ within their current task (travelling, leisure, working, and so on) [11,21].…”

“…The participants' interests are manually speci ed by participants themselves or automatically learned from their manual relevance assessments of the top returned documents in response to their past queries [11,21]. -Topics and tasks: free topics expressed by the participants in situ within their current task (travelling, leisure, working, and so on) [11,21]. -Relevance assessments: each participant who submitted a query (in the diary study), is asked to judge whether a document from the set of top N retrieved results in response to his/her query was relevant or not according to its underlying context.…”