Multiview activity recognition in smart homes with spatio-temporal features

Abstract

Recognizing activities in a home environment is challenging due to the variety of activities that can be performed at home and the complexity of the environment. Multiple cameras are usually needed to cover the whole observation area. This adds camera fusion as another challenge to activity recognition. We propose a hierarchical approach that recognizes both coarse-level and fine-level activities, in which different image features and learning methods are used for different activities based on their characteristics. The paper focuses on discussing the second-level of activity recognition with spatio-temporal features. Specifically, three fusion approaches for multiview activity recognition with spatiotemporal features are presented, including two decision fusion methods and one feature fusion method. They are comparatively analyzed in terms of their tradeoffs on assumptions on system setup, model transferability and recognition rate. Experiments show that challenging activities with subtle motions such as eating, cutting, scrambling, typing, reading etc. can be recognized with our approaches.

A Data Fusion Perspective on Human Motion Analysis Including Multiple Camera Applications

“…The averaging of the multiple features representing pose, global and local motion has been proposed improving the results with respect to other alternatives [15]. A joint Bag-of-Words histogram might be constructed with the local feature descriptors obtained for each one of the views [26], but a higher performance is obtained with other fusion strategies. Projections maximizing the cross-covariance between the R-transform derivatives computed at each view have been defined to learn a joint subspace where the action recognition is performed [12].…”

“…Other proposed measure in the case of employing local features is to choose the camera with the highest number of detections [26]. Different utility measures have been proposed studying the saliency, concavity or variations of silhouette stacks [20].…”

“…This works model the existing correlations among the multiple observations in the structure of the classifier employed for the prediction of the actions. The concatenation of the input features is the most straightforward procedure to perform the fusion [26,15]. The Fused HMM [24] proposes to model correlations among observations coupling the values of the hidden state chains of parallel HMMs defined for each view.…”

Collaborative acquisition of multi-view face images in real-time using a wireless camera network

“…Multi-view camera geometry has been exploited by several recent research efforts to effectively fuse information from different cameras and consequetly improve the accuracy in the context of tasks such as object detection, behavior matching, action classification and reliable foreground extraction [20], [21], [22]. By way of contrast, in this paper we have utilized multi-view geometry to improve the computational efficiency of the system by collaborating among the cameras in real-time and reducing the amount of image processing required.…”

Deep Eye-CU (DECU): Summarization of Patient Motion in the ICU

“…Multi-sensor and multi-camera systems and methods have been applied to smart environments [12,13]. The systems require alterations to existing infrastructure making their deployment in a hospital logistically impossible.…”