Semantic Event Fusion of Different Visual Modality Concepts for Activity Recognition

Abstract

Combining multimodal concept streams from heterogeneous sensors is a problem superficially explored for activity recognition. Most studies explore simple sensors in nearly perfect conditions, where temporal synchronization is guaranteed. Sophisticated fusion schemes adopt problem-specific graphical representations of events that are generally deeply linked with their training data and focused on a single sensor. This paper proposes a hybrid framework between knowledge-driven and probabilistic-driven methods for event representation and recognition. It separates semantic modeling from raw sensor data by using an intermediate semantic representation, namely concepts. It introduces an algorithm for sensor alignment that uses concept similarity as a surrogate for the inaccurate temporal information of real life scenarios. Finally, it proposes the combined use of an ontology language, to overcome the rigidity of previous approaches at model definition, and a probabilistic interpretation for ontological models, which equips the framework with a mechanism to handle noisy and ambiguous concept observations, an ability that most knowledge-driven methods lack. We evaluate our contributions in multimodal recordings of elderly people carrying out IADLs. Results demonstrated that the proposed framework outperforms baseline methods both in event recognition performance and in delimiting the temporal boundaries of event instances.

Long Activity Video Understanding Using Functional Object-Oriented Network

“…Various approaches have been proposed to use knowledge representation for video understanding such as semantic-visual knowledge bases like FrameNet and Imagenet for modeling rich event-centric concepts and their relationships for video event detection [43], a knowledge and probabilistic driven framework for activity recognition [44], semantic representations for event detection [45], [46]. Souza et al deploy objects, actions and their bonds into graphs and use simulated annealing for event inference using temporal connections [47], [48].…”

ChaLearn looking at people: A review of events and resources

“…); Calibration and synchronization of multi-modal data; Multi-modal datasets and evaluation metrics; Leisure, security, health and energy applications based on multi-modal data; Multi-modal Affective Computing and Social Signal processing systems; Multi-modal algorithms designed for GPU, smart phones and game consoles. A total of 17 papers were published withiin this Special Issue at IEEE TPAMI [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47]. 2017 TPAMI Faces: Currently we are organizing a Special Issue at IEEE Transactions on Pattern Analysis and Machine Intelligence journal in the topic of face analysis.…”

Designing a Video Monitoring System for AAL applications: the SPHERE Case Study

“…Considering the 100 houses deployment plan and its financial viability, the low cost consumer RGBD camera, Asus Xtion, was selected. Furthermore, as previously mentioned, RGBD devices now represent the state of the art for indoor activity monitoring [9,20,21,22,11]. For the SPHERE project, the camera needs to be coupled with a machine with suitable processing capacity, minimal intrusion on the user, and minimal cost.…”

“…Video based systems are efficient for implementing alert systems to detect dangerous events like falls, as in [20]. Furthermore, video data analysis allows one to identify specific actions, long term activities, and behavioural patterns [9], with some exploiting contextual information [11]. While video based platforms offer the opportunity to extract unique, continuous, and rich information from the home environment, they also present a number of disadvantages, such as privacy issues [9], user acceptance and system cost and scalability.…”

“…More flexible approaches based on smart cameras and distributed processing have been also employed, for example in [14,15], where colour data was processed in each node and processing results transmitted to central storage nodes. As previously mentioned, RGBD cameras are becoming increasingly more deployed in AAL applications, such as [9,20,27,21,22,11]. For a review of video based AAL systems, the reader is referred to [7,8].…”