A multimodal fusion system for people detection and tracking

Abstract

Because a people detection system that considers only a single feature tends to be unstable, many people detection systems have been proposed to extract multiple features simultaneously. These detection systems usually integrate features using a heuristic method based on the designers' observations and induction. Whenever the number of features to be considered is changed, the designer must change and adjust the integration mechanism accordingly. To avoid this tedious process, we propose a multimodal fusion system that can detect and track people in a scalable, accurate, robust, and flexible manner. Each module considers a single feature and all modules operate independently at the same time. A depth module is constructed to detect people based on the depth-fromstereo method, and a novel approach is proposed to extract people by analyzing the vertical projection in each layer. A color module that detects the human face, and a motion module that detects human movement are also developed. The outputs from these individual modules are fused together and tracked over time, using a Kalman filter. V

Multimodal fusion for multimedia analysis: a survey

“…The authors have also assigned weights to different sensors; however, the determination of these weights has been left to the user. Similar to [40], Yang et al [152] also performed linear weighted fusion of the location information of the objects. However, unlike Foresti and Snidaro [40], Yang et al [152] assigned equal weights to the different modalities.…”

“…However, many of them either have considered equal weights [83,152] or have not elaborated the issue of weight determination [55,67,136], and have left it to the users to decide [40].…”

“…Several researchers have adopted the linear fusion strategy at the feature level for performing various multimedia analysis tasks. Examples include Foresti and Snidaro [40], Yang et al [152] for detecting and tracking people, and Wang et al [136] and Kankanhalli et al [67] for video surveillance and traffic monitoring. The linear fusion strategy has also been adopted at the decision level by several researchers.…”

Human tracking with an infrared camera using a curve matching framework

“…The human body can be tracked by several methods including processing 2D data or 3D positional data by using normal KF [5,6], by segmentation of the main target from the background [7], by multi-sensor fusion data and laser-based leg-detection techniques using on-board laser range finder [8], by techniques such as the hierarchical KF [9], or by the use of quaternions [10]. Humans have also been tracked by tracking multiple features [11], or by localizing the position of the robot itself [12], with respect to its environment. There has been work which deals with detecting and classifying abnormal gaits [13], and human recognition at a distance through gait energy image [14], gait detection using feature extraction from spatial and temporal templates [15], or identification of humans through spatial temporal silhouette analysis [16].…”

“…The weight attached to the KF is w KF = 1 − w CMA (11) where w KF is the weight of the prediction of the KF. It is evident that in a majority of the cases the weight of the KF far outweighs the weight of the CMA.…”