Interaction with mobile applications is often awkward due to the limited and miniaturized input modalities available. This is especially problematic for games where the only incentive to use an application is the pleasure derived from the interaction. It is therefore interesting to examine novel forms of interaction in order to increase the "playability" of mobile games.In this paper we present a simple mobile gaming application on a standard Pocket PC PDA that employs computer vision (CV) as it's main interaction modality. Practical experience with the application demonstrates the feasibility of CV as a primary interaction modality and indicates the high potential of CV as an input modality for mobile devices in the future. Our approach exploits the video capabilities that are becoming ubiquitous on camera equipped smart-phones and PDAs to provide a fun solution for interaction tasks in games like "Pong", "Break-out" or soccer.
Interaction with mobile applications is often awkward due to the limited and miniaturized input modalities available. Kick-Up Menus exploit the video capabilities of camera equipped smart-phones and PDA's to provide a fun solution for interaction tasks like menu selection and simple games.
THE CONCEPTKick-Up Menus can be implemented on video capable mobile devices. The camera facing towards the back of the mobile device ( Fig.1, right) is used to detect "kicking" movements of the user's feet. When a collision between the user's "kick" and an interaction object shown on the screen of the mobile device is detected, a corresponding interaction event for the application is generated.
Figure 1.: Kick-Up-MenusAs shown in Fig.1 interaction objects can be simple menu labels. In these "kick-up-menus" users can open/close submenus or trigger actions by kicking at the corresponding labels. In this case only the horizontal coordinate of the "kick" is used to determine the desired label. The technique is not is not limited to menu selection. The direction and speed of the detected motion can also be exploited.To demonstrate this we have developed a small game called AR-Soccer (Fig.2 ). In the game the player has to kick a virtual ball (a moving interaction object) with his real foot into a virtual goal, bypassing the goalkeeper.
Figure 2: AR-Soccer and the edge-tracking algorithm
IMPLEMENTATIONOur computer vision component calculates the motion of a "kicking object" (e.g. the user's foot) from the incoming video-stream and detects collisions between it and the interaction objects. Since current mobile devices lack highperformance processors we have developed a simple and fast algorithm that combines 2D edge extraction and tracking and operates only in regions of interest (ROI) around the interaction objects [1]. To perform the collision detection, straight edges inside the ROI are vectorized and tracked between two consecutive images (Fig. 2, right). Then the median direction and speed of edge motions inside the ROI is computed. If it points towards an interaction object an interaction event is transmitted to the application OUTLOOK While current hardware limits our system to the processing of 5-7 frames/second, suggesting its use as part of games where the interaction is part of the challenge, feedback from users of the AR-Soccer application has been very positive. We hope that the creation of faster and more robust CV based interaction techniques for productivity applications will become possible on the next generation of PDAs.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.