Breaking the Screen: Interaction Across Touchscreen Boundaries in Virtual Reality for Mobile Knowledge Workers

Abstract

Virtual Reality (VR) has the potential to transform knowledge work. One advantage of VR knowledge work is that it allows extending 2D displays into the third dimension, enabling new operations, such as selecting overlapping objects or displaying additional layers of information. On the other hand, mobile knowledge workers often work on established mobile devices, such as tablets, limiting interaction with those devices to a small input space. This challenge of a constrained input space is intensified in situations when VR knowledge work is situated in cramped environments, such as airplanes and touchdown spaces. In this paper, we investigate the feasibility of interacting jointly between an immersive VR head-mounted display and a tablet within the context of knowledge work. Specifically, we 1) design, implement and study how to interact with information that reaches beyond a single physical touchscreen in VR; 2) design and evaluate a set of interaction concepts; and 3) build example applications and gather user feedback on those applications.

Grand Challenges in Immersive Analytics

“…To solve these challenges, future systems might aim to lower entry barriers to Immersive Analytics interaction. One promising possibility is to combine established interfaces, like keyboard, mouse or tablet input, with immersive HMDs [11,14,54,58,135]. This is closely related to the question of discoverability: many "natural" interaction techniques used in Immersive Analytics, including gestures, speech, and tangible tokens, are not obvious to users [98], which risks hiding possible actions from them.…”

PoVRPoint: Authoring Presentations in Mobile Virtual Reality

“…As people work more remotely, from touchdown offices, on the go or from home, the importance of mobility and privacy increases. This paper, and other prior works [4,15] show that this mode of work can be more efficient than the current use of 2D displays, as HMDs can fill the user's full field of view with potentially very large virtual displays and show world grounded stereoscopic information, which the user can interact with directly.…”

“…Virtual Reality (VR) as an instance of Extended Reality (XR), can benefit users immensely, as it provides a much larger display space than traditional mobile devices, independent of the physical environment and allows rendering of 3D elements above the screen, potentially mitigating effects of a small physical screen. [4,15,18]. HMDs can also increase privacy and reduce environmental clutter [18,57].…”

“…HMDs can also increase privacy and reduce environmental clutter [18,57]. Using VR to support knowledge workers on the go, and therefore in limited spaces, has already been discussed in previous work [18,43,77] and interaction techniques for such scenarios have been developed for multi-screen environments [4] or spreadsheet applications [15]. This approach is also supported by the advancement of new hardware, as it is already possible to use inside-out tracking HMDs on the go and multiple manufacturers work on light HMDs as a replacement for physical screens.…”

“…As hand and eye gaze tracking technologies are already being incorporated into commercial HMDs this opens up new interaction possibilities. Along with mobile devices such as tablets, touch-based interaction can be augmented with spatial interaction above and around the screen [4,15] to facilitate knowledge workers' tasks and potentially improve the overall interaction experience in limited spaces. Therefore, we combine a VR HMD with eye-tracking and bimanual pen and touch techniques, where pen-based input can be sensed on and above mobile devices, requiring less interaction space [15].…”