The development and evaluation of a survey to measure user engagement

Abstract

Facilitating engaging user experiences is essential in the design of interactive systems. To accomplish this, it is necessary to understand the composition of this construct and how to evaluate it. Building on previous work that posited a theory of engagement and identified a core set of attributes that operationalized this construct, we constructed and evaluated a multidimensional scale to measure user engagement. In this paper we describe the development of the scale, as well as two large-scale studies (N = 440 and N = 802) that were undertaken to assess its reliability and validity in online shopping environments. In the first we used Reliability Analysis and Exploratory Factor Analysis to identify six attributes of engagement: Perceived Usability, Aesthetics, Focused Attention, Felt Involvement, Novelty, and Endurability. In the second we tested the validity of and relationships among those attributes using Structural Equation Modeling. The result of this research is a multidimensional scale that may be used to test the engagement of software applications. In addition, findings indicate that attributes of engagement are highly intertwined, a complex interplay of user-system interaction variables. Notably, Perceived Usability played a mediating role in the relationship between Endurability and Novelty, Aesthetics, Felt Involvement, and Focused Attention.

Analyzing the Role of Interactivity in User Experience

“…Lee (2005) To summarise, interactivity has been investigated in several studies (Cyr, Head & Ivanov 2009;Cyr, Head, Larios & Pan, 2009;O'Brien 2010;O'Brien & Toms 2010;Teo et al 2003) which extended TAM measures (Venkatesh et al 2003), so whereas the positive influence of interactive features on efficiency, effectiveness and satisfaction has been demonstrated, the connection between interactivity and affect and the non-instrumental aspects of UX (e.g. aesthetics) has received less attention.…”

User-controllable personalization: A case study with SetFusion

“…How does controllability affect user engagement with a recommender system? This question was motivated by an existent gap between the research on user engagement in software applications that highly recommends the use of both subjective and objective metrics (O'Brien & Toms, 2010) and the results of previous research (Hijikata et al, 2012;B. P. Knijnenburg, S. Bostandjiev, et al, 2012), that support the effect of controllability on user engagement, mainly by considering subjective measures (surveys) and one or very few objective metrics (average rating).…”

Task complexity, vertical display and user interaction in aggregated search

“…The sub-scales from O'Brien's Engagement Scale [21,22] formed the basis of the sub-scales we used in this study to elicit subjects' system evaluations. The Engagement Scale [22] consists of a 31-item scale with 6 sub-scales which measure the following aspects of engagement (number in parenthesis indicates how many items are on each sub-scale): Focused Attention (7), Perceived Usability (8), Endurability (5), Novelty (3), Aesthetics (5) and Felt Involvement (3).…”

“…The Engagement Scale [22] consists of a 31-item scale with 6 sub-scales which measure the following aspects of engagement (number in parenthesis indicates how many items are on each sub-scale): Focused Attention (7), Perceived Usability (8), Endurability (5), Novelty (3), Aesthetics (5) and Felt Involvement (3). For all scales, subjects respond by indicating their level of agreement with the items (1=strongly disagree; 5=strongly agree).…”