Contextual information access with Augmented Reality

Abstract

We have developed a prototype platform for contextual information access in mobile settings. Objects, people, and the environment are considered as contextual channels or cues to more information. The system infers, based on gaze, speech and other implicit feedback signals, which of the contextual cues are relevant, retrieves more information relevant to the cues, and presents the information with Augmented Reality (AR) techniques on a handheld or headmounted display. The augmented information becomes potential contextual cues as well, and its relevance is assessed to provide more information. In essence, the platform turns the real world into an information browser which focuses proactively on the information inferred to be the most relevant for the user. We present the first pilot application, a Virtual Laboratory Guide, and its early evaluation results.

G-VOILA: Gaze-Facilitated Information Querying in Daily Scenarios

“…With the advancement of Augmented Reality (AR) devices, researchers have integrated IR closely with physical world through Head-Mounted Displays (HMDs) [3,14,27,39,52,71,76], which is defined as reality-based information retrieval (RBIR) by [15]. RBIR adheres to the principle of "find what you need with zero query terms or less" [7], adopting a proactive strategy to impose information on certain objects in order to reduce inquiry costs [27,39,76].…”

“…To tackle information overload in AR, Piening et al [52] proposed gaze-adaptive interfaces to modulate the display of information panels. Meanwhile, researchers [3,14,15] introduced multi-round IR operations leveraging AR's interactivity to facilitate precise information acquisition, including selection and search refinement. For example, a user interested in avocados at a grocery store may select "nutrition details" rather than "recipes" on avocados' information panel.…”

“…Current methods for gaze-based intent prediction typically employ deep learning techniques, such as CNNs, for predefined classifications [2,57]. Gaze data also serve as key indicator of interest in RBIR [3,52]. To date, no research has integrated eye movements with natural language to deduce user intent.…”

“…As the cyber and physical spaces quickly merge, people exhibit a significant demand for information retrieval (IR) anywhere and anytime in their daily lives [10,14,15,22,56], no longer confined to a specific device or location. With advancements in the computational capabilities of wearable devices, the incorporation of a virtual assistant that can provide on-demand, in-situ answers to users' inquiries has the potential to greatly facilitate the interaction with surrounding targets [68,75] and enhance the naturalness of the user's information retrieval experience [3]. Specifically, smart glasses with gaze tracking open new possibilities for natural information retrieval techniques in daily scenarios by combining the voice and gaze modalities [35,62].…”

“…Prior research has investigated the potential of utilizing smart glasses [29,67] equipped with gaze tracking to streamline information retrieval related to physical objects [3,15,26,39,52,71,74]. These approaches enhanced information retrieval by superimposing digital widgets around physical entities, allowing users to access information through interacting with the widgets in a multi-stage manner [3,14]. While this has been a significant stride, the ultimate goal is to enable users to retrieve information using natural expressions, eliminating the need for these intermediary widgets.…”