Gestures Everywhere

Abstract

Gestures Everywhere is a dynamic framework for multimodal sensor fusion, pervasive analytics and gesture recognition. Our framework aggregates the real-time data from approximately 100 sensors that include RFID readers, depth cameras and RGB cameras distributed across 30 interactive displays that are located in key public areas of the MIT Media Lab. Gestures Everywhere fuses the multimodal sensor data using radial basis function particle filters and performs real-time analysis on the aggregated data. This includes key spatio-temporal properties such as presence, location and identity; in addition to higher-level analysis including social clustering and gesture recognition. We describe the algorithms and architecture of our system and discuss the lessons learned from the systems deployment.

EagleSense

“…However, since it readily provides real-time tracking of users, devices, and activities, we expect it to become a part of a larger tracking ecosystem consisting of multiple cameras and sensors. For example, this could advance strategies such as those in the Gestures Everywhere framework [12] which integrates sensing across multiple spaces to provide both low-and high-level tracking information about the users and groups. A generic and ubiquitous infrastructure for ad-hoc interactions, by combining EagleSense and other space models (e.g., [32,52]), can enable visualizations of interactive spaces as well as opting-in and opting-out interaction techniques at scale.…”

Audience monitor

“…To understand the interaction of viewers across several displays, Gillian et al [13] designed and deployed a framework that is capable of recognizing viewers across displays (using both depth-cameras and additional sensing functionality). While this system provides analytical insights into how viewers interact with content, it also provides additional features to the user and personalized content across a range of displays.…”