A Scalable Approach to Activity Recognition based on Object Use

Abstract

We propose an approach to activity recognition based on detecting and analyzing the sequence of objects that are being manipulated by the user. In domains such as cooking, where many activities involve similar actions, object-use information can be a valuable cue. In order for this approach to scale to many activities and objects, however, it is necessary to minimize the amount of human-labeled data that is required for modeling. We describe a method for automatically acquiring object models from video without any explicit human supervision. Our approach leverages sparse and noisy readings from RFID tagged objects, along with common-sense knowledge about which objects are likely to be used during a given activity, to bootstrap the learning process. We present a dynamic Bayesian network model which combines RFID and video data to jointly infer the most likely activity and object labels. We demonstrate that our approach can achieve activity recognition rates of more than 80% on a real-world dataset consisting of 16 household activities involving 33 objects with significant background clutter. We show that the combination of visual object recognition with RFID data is significantly more effective than the RFID sensor alone. Our work demonstrates that it is possible to automatically learn object models from video of household activities and employ these models for activity recognition, without requiring any explicit human labeling.

Unstructured human activity detection from RGBD images

“…Most previous work on activity classification has focused on using 2D video (e.g., [26,10]) or RFID sensors placed on humans and objects (e.g., [41]). The use of 2D videos leads to relatively low accuracy (e.g., 78.5% in [19]) even when there is no clutter.…”

“…One common approach is to use space-time features to model points of interest in video [15,6]. Several authors have supplemented these techniques by adding more information to these features [11,40,41,19,25,30]. However, this approach is only capable of classifying, rather than detecting, activities.…”

A study on automatic recognition of object use exploiting motion correlation of wireless sensors

“…A common solution is to detect whether a particular user, or rather his arm, is in close proximity to an object by employing the RFID [4,8,31] technology or other RFbased solutions [6]. Each object is equipped with an RFID tag, while the user wears a reader on his hand or lower arm.…”

“…Each of these has its own limitations. RFID systems may erroneously mark an object as being held by the user just because it is in close proximity or, conversely, the user's interaction may be missed when the object is grabbed at a great distance from the RFID tag [31]. The techniques using contact switches and monitoring the power consumption of appliances in the home provide no information on the user's identity and therefore those only provide a suitable solution when the identity of the user either is known implicitly or not important.…”

“…An essential component in this vision is the ability of detecting which objects are being used at any moment, in a non-intrusive manner. A number of solutions have already been proposed in the literature: RFID [4,21,22,31], contact switches [26,27] and power consumption monitoring of electrical appliances in the home [3]. Each of these has its own limitations.…”