Metaphoraction: Support Gesture-based Interaction Design with Metaphorical Meanings

Abstract

Previous user experience research emphasizes meaning in interaction design beyond conventional interactive gestures. However, existing exemplars that successfully reify abstract meanings through interactions are usually case-specific, and it is currently unclear how to systematically create or extend meanings for general gesture-based interactions. We present Metaphoraction, a creativity support tool that formulates design ideas for gesture-based interactions to show metaphorical meanings with four interconnected components: gesture , action , object , and meaning . To represent the interaction design ideas with these four components, Metaphoraction links interactive gestures to actions based on the similarity of appearances, movements, and experiences; relates actions to objects by applying the immediate association; bridges objects and meanings by leveraging the metaphor TARGET-SOURCE mappings. We build a dataset containing 588,770 unique design idea candidates through surveying related research and conducting two crowdsourced studies to support meaningful gesture-based interaction design ideation. Five design experts validate that Metaphoraction can effectively support creativity and productivity during the ideation process. The paper concludes by presenting insights into meaningful gesture-based interaction design and discussing potential future uses of the tool.

Beyond Numbers: Creating Analogies to Enhance Data Comprehension and Communication with Generative AI

“…proposed using metaphor's target-source concepts to facilitate visual design ideation [23] and gesture design [50] imaginatively. To support science writers in explaining unfamiliar concepts, Kim et al [25] launched Metaphorian, which facilitates the search, extension, and iterative revision of extended metaphors.…”

“…We adapted the methods used by Sun et al [50], in which a crowdsourcing task is designed to collect participants' feedback on their understanding of autogenerated data. In general, our study aims to collect participants' responses regarding three perspectives: their understanding of the given data, the extent of measurement of the data scale, and the degree of engagement with the context.…”

Towards open-set text recognition via label-to-prototype learning

“…In recent years, the study of human gestures based on cameras and sensors has gained increasing significance in human-machine interaction (HMI) [1], [2], [3]. In this interaction paradigm, gesture recognition serves as a foundational task and has been extensively applied across diverse domains, such as rehabilitation robotics [4], [5], mobile interaction [6], [7], virtual reality [8] and exoskeleton control [9].…”

“…* Corresponding author. 1 This work was partially supported by OYMotion Technologies. 2 Chen Liu, Can Han, Chengfeng Zhou, Crystal Cai, Suncheng Xiang and Dahong Qian are affiliated with the School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China.…”

Gesture-Based Interactions: Integrating Accelerometer and Gyroscope Sensors in the Use of Mobile Apps

“…Gesture-based interactions. This involves interacting with the UI through motion/ges ture detection, and is perhaps the least implemented application in mobile apps [12,13].…”