Extending Keyboard Shortcuts with Arm and Wrist Rotation Gestures

Buschek, Daniel; Roppelt, Bianka; Alt, Florian

doi:10.1145/3173574.3173595

Cited by 17 publications

(7 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There have been several approaches in extending the basic input capabilities of physical keyboard beyond individual button presses. Specifically, input on, above and around the keyboard surface have been proposed using acoustic [38,43], pressure [17,52,81], proximity [74], capacitive sensors [5,21,32,64,71,76], cameras [39,62,80], body-worn orientation sensors [9] or even unmodified physical keyboards [46,82]. Besides sensing, actuation of keys has also been explored [4].…”

Section: Augmenting Physical Keyboardsmentioning

confidence: 99%

ReconViguRation: Reconfiguring Physical Keyboards in Virtual Reality

Schneider

Otte

Gesslein

et al. 2019

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

Figure 1: Example for reconfiguring the input and output space of individual keys of a physical keyboard in virtual reality. Top row from left to right: emoji entry, special characters, secure password entry using randomized keys. Bottom row from left to right: foreign languages, browser shortcuts, text processing macros. ABSTRACTPhysical keyboards are common peripherals for personal computers and are efficient standard text entry devices. Recent research has investigated how physical keyboards can be used in immersive headmounted display-based Virtual Reality (VR). So far, the physical layout of keyboards has typically been transplanted into VR for replicating typing experiences in a standard desktop environment. In this paper, we explore how to fully leverage the immersiveness of VR to change the input and output characteristics of physical keyboard interaction within a VR environment. This allows individual physical keys to be reconfigured to the same or different actions and visual output to be distributed in various ways across the VR representation of the keyboard. We explore a set of input and output mappings for reconfiguring the virtual presentation of physical keyboards and probe the resulting design space by specifically designing, implementing and evaluating nine VR-relevant applications: emojis, languages and special characters, application shortcuts, vir-* tual text processing macros, a window manager, a photo browser, a whack-a-mole game, secure password entry and a virtual touch bar. We investigate the feasibility of the applications in a user study with 20 participants and find that, among other things, they are usable in VR. We discuss the limitations and possibilities of remapping the input and output characteristics of physical keyboards in VR based on empirical findings and analysis and suggest future research directions in this area.

show abstract

Section: Augmenting Physical Keyboardsmentioning

confidence: 99%

ReconViguRation: Reconfiguring Physical Keyboards in Virtual Reality

Schneider

Otte

Gesslein

et al. 2019

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

show abstract

“…GestAKey allowed for multiple actions being associated to a keystroke through touch sensitive keycaps [93], an approach that has repeatedly been explored [92,116]. Other gesture approaches for extending keyboard shortcuts [14] such as gesturing on/above the surface of the keyboard have also been examined [100,113]. More recently, touch sensitivity has been explored for keyboards in AR/MR [80], suggesting new possibilities for per-key states driven by hand tracking technology in particular.…”

Section: Supporting Interactionsmentioning

confidence: 99%

“…FingerArc and FingerChord [114,115] could also be enacted entirely using MR headset sensing, with visual guidance now rendered in situ rather than on a separate display off-device [30]. Other gesture approaches for extending keyboard shortcuts [14] and gesturing on/above the surface of the keyboard [100,113] could equally be supported with congruent visuals, effectively democratising access to a variety of previously envisaged novel keyboard interactions.…”

Section: Touch-displaymentioning

confidence: 99%

Creating and Augmenting Keyboards for Extended Reality with the K eyboard A ugmentation T oolkit

McGill

Brewster

Medeiros

et al. 2022

ACM Trans. Comput.-Hum. Interact.

View full text Add to dashboard Cite

This article discusses the Keyboard Augmentation Toolkit (KAT), which supports the creation of virtual keyboards that can be used both for standalone input (e.g., for mid-air text entry) and to augment physically tracked keyboards/surfaces in mixed reality. In a user study, we firstly examine the impact and pitfalls of visualising shortcuts on a tracked physical keyboard, exploring the utility of virtual per-keycap displays. Supported by this and other recent developments in XR keyboard research, we then describe the design, development, and evaluation-by-demonstration of KAT. KAT simplifies the creation of virtual keyboards (optionally bound to a tracked physical keyboard) that support enhanced display —2D/3D per-key content that conforms to the virtual key bounds; enhanced interactivity —supporting extensible per-key states such as tap, dwell, touch, swipe; flexible keyboard mappings that can encapsulate groups of interaction and display elements, e.g., enabling application-dependent interactions; and flexible layouts —allowing the virtual keyboard to merge with and augment a physical keyboard, or switch to an alternate layout (e.g., mid-air) based on need. Through these features, KAT will assist researchers in the prototyping, creation and replication of XR keyboard experiences, fundamentally altering the keyboard’s form and function.

show abstract

“…Stančić et al presented a hand gesture-based human-robot interface for navigating and controlling a mobile robot in an indoor or outdoor environment [20]. Acceleration signals can also be incorporated with gyroscope outputs to provide an interactive way for interfacing with smartwatches or computers using finger gestures [21], wrist rotation [22], and hand motion such as waving [23].…”

Section: Wearable Sensorsmentioning

confidence: 99%

Are You Wearing a Mask? Detecting If a Person Wears a Mask Using a Wristband

Msigwa

Baek

Bernard

et al. 2022

Sensors

View full text Add to dashboard Cite

Coronavirus 2019 (COVID-19) has posed a serious threat to the lives and health of the majority of people worldwide. Since the early days of the outbreak, South Korea’s government and citizens have made persistent efforts to provide effective prevention against further spread of the disease. In particular, the participation of individual citizens in complying with the necessary code of conduct to prevent spread of the infection, through measures such as social distancing and mask wearing, is as instrumental as the geographical tracking of the trajectory of the infected. In this paper, we propose an activity recognition method based on a wristband equipped with an IR array and inertial measurement unit (IMU) to detect individual compliance with codes of personal hygiene management, such as mask wearing, which are recommended to prevent the spread of infectious diseases. The results of activity recognition were comparatively analyzed by applying conventional machine learning algorithms and convolutional neural networks (CNNs) to the IMU time series and IR array thermal images collected from 25 subjects. When CNN and 24 × 32 thermal images were used, 97.8% accuracy was achieved (best performance), and when 6 × 8 low-resolution thermal images were used, similar performance with 97.1% accuracy was obtained. In the case of using IMU, the performance of activity recognition was lower than that obtained with the IR array, but an accuracy of 93% was achieved even in the case of applying machine learning algorithms, indicating that it is more suitable for wearable devices with low computational capability.

show abstract

Extending Keyboard Shortcuts with Arm and Wrist Rotation Gestures

Cited by 17 publications

References 20 publications

ReconViguRation: Reconfiguring Physical Keyboards in Virtual Reality

ReconViguRation: Reconfiguring Physical Keyboards in Virtual Reality

Creating and Augmenting Keyboards for Extended Reality with the K eyboard A ugmentation T oolkit

Are You Wearing a Mask? Detecting If a Person Wears a Mask Using a Wristband

Contact Info

Product

Resources

About