Multimodal Integration of Micro-Doppler Sonar and Auditory Signals for Behavior Classification With Convolutional Networks

Abstract

The ability to recognize the behavior of individuals is of great interest in the general field of safety (e.g. building security, crowd control, transport analysis, independent living for the elderly). Here we report a new real-time acoustic system for human action and behavior recognition that integrates passive audio and active micro-Doppler sonar signatures over multiple time scales. The system architecture is based on a six-layer convolutional neural network, trained and evaluated using a dataset of 10 subjects performing seven different behaviors. Probabilistic combination of system output through time for each modality separately yields 94% (passive audio) and 91% (micro-Doppler sonar) correct behavior classification; probabilistic multimodal integration increases classification performance to 98%. This study supports the efficacy of micro-Doppler sonar systems in characterizing human actions, which can then be efficiently classified using ConvNets. It also demonstrates that the integration of multiple sources of acoustic information can significantly improve the system's performance.

A Radar-Based Smart Sensor for Unobtrusive Elderly Monitoring in Ambient Assisted Living Applications

“…Range sensing is another contactless ambient modality based on the remote measurement of distances. Commonly employed sensors are pyroelectric infrared (PIR) [ 38 , 39 , 40 ], sonar [ 41 ], Lidar/range camera [ 42 , 43 ], radar [ 44 ] and Wi-Fi systems [ 45 ]. Acceptability and performance are quite good, especially in the case of Range camera and Radar, since depth maps and radar scans are not able to capture privacy-sensitive information.…”

Using Artificial Intelligence to Achieve Auxiliary Training of Table Tennis Based on Inertial Perception Data

“…According to the different types of data used for human action recognition, human action recognition is usually divided into the following three main categories: human action recognition based on vision, human action recognition based on acoustics, and human action recognition based on inertial sensors [ 13 ]. Visual action recognition extracts human action data from image or video data obtained by optical sensors [ 14 , 15 ], acoustic action recognition uses sound signals for high-precision hand action tracking and gesture recognition [ 16 , 17 ], inertial sensor action recognition focuses on extracting human action inertial data from wearable inertial sensors [ 18 , 19 ]. Liu Yutao [ 20 ] summarized the three methods, as shown in the following Table 1 .…”

Acoustic micro‐Doppler signal processing with foveated electronic cochlea

“…If the object itself contains moving parts, each moving part will result in a modulation of the base Doppler frequency shift, known as the micro‐Doppler effect. A micro‐Doppler sonar was reported in [1] to produce signal signatures of human gait and subsequently employed in behaviour analysis and classification [2].…”

“…If the object itself contains moving parts, each moving part will result in a modulation of the base Doppler frequency shift, known as the micro-Doppler effect. A micro-Doppler sonar was reported in [1] to produce signal signatures of human gait and subsequently employed in behaviour analysis and classification [2].The sonar micro-Doppler system in [1] employs principles also found in echolocating animals such as bats and dolphins that emit mechanical waves that bounce onto objects in an environment and allow them to place themselves in space, navigate and even hunt [3,4]. Echolocating bats can fall into two different categories, the ones emitting a frequency-modulated ultrasound signal, known as FM, and the ones emitting a constant frequency signal, known as CF.…”