PROMETHEUS: heterogeneous sensor database in support of research on human behavioral patterns in unrestricted environments

Abstract

The multi-modal multi-sensor PROMETHEUS database was created in support of research and development activities [PROMETHEUS (FP7-ICT-214901): http://www.prometheus-FP7.eu] aiming at the creation of a framework for monitoring and interpretation of human behaviors in unrestricted indoor and outdoor environments. The distinctiveness of the PROMETHEUS database comes from the unique sensor sets, used in the various recording scenarios, but also from the database design, which covers a range of real-world applications, correlated to smart-home automation and indoors/outdoors surveillance of public areas. Numerous single-person and multi-person scenarios, but also scenarios with interactions between groups of people, motivated by these applications were implemented with the help of skilled actors and supernumerary personnel. In these scenarios, the actors and personnel were instructed to implement a range of typical and atypical behaviors, and simulations of emergency and crisis situations. In summary, the database contains more than 4 h of synchronized recordings from heterogeneous sensors (an infrared motion detection sensor, thermal imaging cameras, overview/surveillance video cameras, close-view video cameras, a 3D camera, a stereoscopic camera, a general-purpose camcoder, microphone arrays, and motion capture equipment) collected in common setups, simulating smart-home environment, airport, and ATM security environment. Selected scenes of the database were annotated for the needs of human detection and tracking. The entire audio part of the database was annotated for the needs of sound event detection, sound source enumeration, emotion recognition, etc

HMM-Based Abnormal Behaviour Detection Using Heterogeneous Sensor Network

“…As mentioned, the ATM scenarios of PROMETHUS [8] multimodal database are used in this paper. There are four selected ATM scenes with different durations.…”

“…1. A superimposed view of the ATM scenario when a rubbery is happening multimodal database, referred as PROMETHEUS database 1 [8]. This database is in support of the development and the evaluation of the algorithms which are intended to analyze and identify human actions and behaviors in the context of surveillance using multi-modal approaches.…”

Automatic detection of cow/calf vocalizations in free-stall barn

“…Hence, captured audio is influenced by factors that could not be controlled such as rain, snow, wind, noise, interferences from background activities, etc. All such events were kept in the dataset as our target is to process the raw recordings without any significant efforts on manual annotation [23]. Indeed, data labeling was carried out only for the first 6 hours of recordings (environmental noise, and animal vocalizations) based on the outcome of Hilbert follower as described in section II-A.…”

ASMD: an automatic framework for compiling multimodal datasets with audio and scores

“…The framework is complemented with a Python API written in Cython. 3 It allows in particular to load various dataset definitions aside of the official ones. The API provides methods to retrieve audio and annotations in various structures, such as a matrix list of notes similar to the one used by Matlab MIDI Toolbox [23] Moreover, since the API basically consists in a class representing a large dataset, it is very easy to extend it in order to use it in conjunction of PyTorch or TensorFlow frameworks for training neural network models.…”

“…A recent trend in computer science is the adoption of multimodal strategies for increasing the effectiveness of algorithmic solutions in several domains [1][2][3][4][5]. This comes as a natural consequence of the a) ever-increasing availability of computational resources, which are now able to deal with big data, and b) popularity of machine learning algorithms, the performance of which is boosted as more data (including multimodal) becomes available.…”