Examining the generalizability of the User Engagement Scale (UES) in exploratory search

Abstract

The user experience is an integral component of interactive information retrieval (IIR). However, there is a twofold problem in the measurement of user experience. Firstly, while many IIR studies have relied on a single dimension of user feedback, that of satisfaction, experience is a much more complex concept. IIR in general, and exploratory search more specifically, are dynamic, multifaceted experiences that evoke pragmatic and hedonic needs, expectations, and outcomes that are not adequately captured by user satisfaction. Secondly, questionnaires, which are typically the means in which user's attitudes and perceptions are measured, are not typically subjected to rigorous reliability and validity testing. To address these issues, we administered the multidimensional User Engagement Scale (UES) in an exploratory search environment to assess users' perceptions of the Perceived Usability (PUs), Aesthetics (AE), Novelty (NO), Felt Involvement (FI), Focused Attention (FA), and Endurability (EN) aspects of the experience. In a typical laboratory-style study, 381 participants performed three relatively complex search tasks using a novel search interface, and responded to the UES immediately upon completion. We used Principal Axis Factor Analysis and Multiple Regression to examine the factor structure of UES items and the relationships amongst factors. Results showed that three of the six sub-scales (PUs, AE, FA) were stable, while NO, FI and EN merged to form a single factor. We discuss recommendations for revising and validating the UES in light of these findings.

The Relationship Between User Perception and User Behaviour in Interactive Information Retrieval Evaluation

“…However, our results identified four factors, which is consistent with the result obtained from the wikiSearch study [17] and Facebook [4]. This suggests that in a searching environment, the dimensions of UES structure may remain consistent regardless of data type (text data or image data), or perhaps it is due to the presence of rich information and interactivity.…”

“…The reliability analysis of all six original UES sub-scales demonstrated good internal consistency, which aligns with previous studies [16,17]. In our correlation analysis, Perceived Usability had a positive and moderate relationship with Focused Attention, which is in contrast to the results of the wikiSearch study, which found a negative correlation between the two [16].…”

“…Unlike other measures, the model underpinning the UES shows how Endurability is explained either directly or indirectly by the other five dimensions. The UES has been used to evaluate multiple types of systems (e.g., e-shopping [16], wikiSearch [17], Facebook [4]). This scale follows standard psychometric scale development methods [7], and has been tested for reliability and validity.…”

“…This scale follows standard psychometric scale development methods [7], and has been tested for reliability and validity. Although differences have emerged [17] in the various applications, it is the most tested measure of user perception of a system.…”

From Acceptability to Acceptance: Does Experience with the Product Influence User Initial Representations?

“… Assessment of UX factors: Usefulness (4 items), Usability (4 items), Stimulation (4 items) and Trust (3 items)  Assessment of affective-factors: Motivation (5 items), Self-image (4 items) and Social Influence (3 items)  Assessment of acceptability/acceptance: Intention to use (4 items) Some items were constructed specifically for this study and others were based on existing questionnaires as UTAUT [3], Attrackdiff [8] and Engagement Scale [17]. For each item, participants had to answer on an 11 points likert scale (from 0 to 10).…”