The evaluation of adaptive and personalised information retrieval systems: a review

Abstract

A current problem with the research of adaptive systems is the inconsistency of evaluation applied to the adaptive systems. However, evaluating an adaptive system is a difficult task due to the complexity of such systems. Evaluators need to ensure correct evaluation methods and measurement metrics are used. This paper reviews a variety of evaluation techniques applied in adaptive and user-adaptive systems. More specifically, it focuses on the user-centred evaluation of adaptive systems such as personalised recommender systems and adaptive information retrieval systems. The review tackles the question of 'How have user-centred evaluations of adaptive and user-adaptive systems been conducted and how can these evaluation practices be improved?' Based on the analysed results of the: (a) evaluation approaches, (b) user-centred evaluation techniques, and (c) evaluation metrics, we propose an evaluation framework for end-user experience in evaluating adaptive systems (EFEx).

Modelling Spatio-Temporal Relevancy in Urban Context-Aware Pervasive Systems Using Voronoi Continuous Range Query and Multi-Interval Algebra

“…Each route was traversed by a visitor equipped with a laptop, a GPS and the software designed based on the proposed model. This paper evaluates the results of the experiments based on three parameters namely; the accuracy of the results [9], the performance of the model [12] and the satisfaction of the users [13].…”

“…The user's satisfaction with the services provided is a key issue in modeling context-aware systems [11,13]. It is obviously crucial for adoption and acceptance of such technologies [12,39].…”

“…It is obviously crucial for adoption and acceptance of such technologies [12,39]. This study considered seven elements for estimating user satisfaction from the model and context-aware services, namely: (1) the usefulness of the prediction in the user's decision making process for choosing his/her related contexts [2,13], (2) the usefulness of the presentation of contexts that have spatial relationships behind the user and in the opposite direction, (3) the effect of the contexts introduction based on the velocity (this factor is derived from the "adapt to varying situations of the user" parameter pointed by [12], (4) the order of the contexts presentation and (5) the responding time of the algorithm to the user [3,12], (6) number of navigation steps [13], and (7) task success [13].…”

“…Furthermore the satisfaction of the users by using VCRQ in the first step of the proposed algorithm leas to the 2% increase rather than when we used DRNQ [33]. Another parameter which is important in context-aware system is usability of the model [13] STRM-CAS have 4 main components for modeling spatio-temporal relevancy (DSI, SI, TIU and TIC) and it is possible for every navigation applications to adapt with defining them. Indeed only the related contexts are changed based on the user preferences and goals.…”

Learning Personalization Based on Learning Style instruments

“…In order to evaluate the impact of using the LAES system on student performance, an experimental evaluation approach was used. This approach is recommended by several researchers in this field [19].…”