Multi-Modal Cross Learning for Improved People Counting using Short-Range FMCW Radar

Aydogdu, Cem Yusuf; Hazra, Souvik; Santra, Avik; Weigel, Robert

doi:10.1109/radar42522.2020.9114871

Cited by 24 publications

(11 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Various applications rely on multi-modal sensory input as shown in [5]- [14]. A multi-modal approach for gesture recognition using video, depth camera, and optical flow is shown in [6].…”

Section: Related Workmentioning

confidence: 99%

“…Stacked autoencoders are used in [14] for a cross-modal audio mapping and a gesture-to-audio conversion task. One example for a radar sensor in combination with a camera for people counting is shown in [5]. This approach uses 60 GHz radar data in form of range-angle images and the camera images are preprocessed to density heatmaps.…”

Section: Related Workmentioning

confidence: 99%

“…However, there are niche applications and also niche sensor data types like radar or thermal images without publicly available datasets or pretrained neural networks (models). Several Deep Learning approaches for radar focus on human gait classification [3], gesture recognition [4], or counting people [5] with great success. But there are no large generalized datasets and pretrained models covering all kinds of radar applications and sensors like ImageNet [1] for image classification.…”

Section: Introductionmentioning

confidence: 99%

“…4) A cooktop occupancy detection is shown in this paper. 5) We believe that radars are preferable to cameras for privacy reasons and acceptance when used in private households. The advantages and disadvantages of the three sensor types radar, thermal camera, and RGB camera for the application are discussed and evaluated in this paper.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Multi-Modal Cross Learning for an FMCW Radar Assisted by Thermal and RGB Cameras to Monitor Gestures and Cooking Processes

et al. 2021

View full text Add to dashboard Cite

This paper proposes a multi-modal cross learning approach to augment the neural network training phase by additional sensor data. The approach is multi-modal during training (i.e., radar Range-Doppler maps, thermal camera images, and RGB camera images are used for training). In inference, the approach is single-modal (i.e., only radar Range-Doppler maps are needed for classification). The proposed approach uses a multi-modal autoencoder training which creates a compressed data representation containing correlated features across modalities. The encoder part is then used as a pretrained network for the classification task. The benefits are that expensive sensors like high resolution thermal cameras are not needed in the application but a higher classification accuracy is achieved because of the multi-modal cross learning during training. The autoencoders can also be used to generate hallucinated data of the absent sensors. The hallucinated data can be used for user interfaces, a further classification, or other tasks. The proposed approach is verified within a simultaneous cooking process classification, 2x2 cooktop occupancy detection, and gesture recognition task. The main functionality is an overboil protection and gesture control of a 2x2 cooktop. The multi-modal cross learning approach considerably outperforms single-modal approaches on that challenging classification task.

show abstract

“…Various applications rely on multi-modal sensory input as shown in [5]- [14]. A multi-modal approach for gesture recognition using video, depth camera, and optical flow is shown in [6].…”

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Multi-Modal Cross Learning for an FMCW Radar Assisted by Thermal and RGB Cameras to Monitor Gestures and Cooking Processes

et al. 2021

View full text Add to dashboard Cite

show abstract

“…In this application, the complementary nature of radar and optical data is not used per se (the image registration is based on features that are present in both modalities). Only in recent years, methods exploiting the complementary nature of radar and one or more other modalities have been reported [14][15][16][17][18]. Some of these methods are particularly focussed on data fusion and less on cross learning, that is, all modalities are always assumed to be present [14,15].…”

Section: Introductionmentioning

confidence: 99%

Classification of human activity using radar and video multimodal learning

Jong¹,

Wit²,

Uysal

2021

IET Radar Sonar & Navi

View full text Add to dashboard Cite

In the defence and security domain, camera systems are widely used for surveillance. The major advantage of using camera systems for surveillance is that they provide highresolution imagery, which is easy to interpret. However, the use of camera systems and optical imagery has some drawbacks, especially for application in the military domain. In poor lighting conditions, dust or smoke the image quality degrades and, additionally, cameras cannot provide range information too. These drawbacks can be mitigated by exploiting the strengths of radar. Radar performance can be largely maintained during the night, in various weather conditions and in dust and smoke. Moreover, radar provides the distance to detected objects. Since, the strongpoints and weaknesses of radar and camera systems seem complementary, a natural question is: can radar and camera systems learn from each other? Here the potential of radar/video multimodal learning is evaluated for human activity classification. The novelty of this work is the use of radar spectrograms and related video frames for classification with a multimodal neural network. Radar spectrograms and video frames are both two-dimensional images, but the information they contain is of different nature. This approach was adopted to limit the required preprocessing load, while maintaining the complementary nature of the sensor data.This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

show abstract

Cross‐Modal Learning

2022

Methods and Techniques in Deep Learning

View full text Add to dashboard Cite

Multi-Modal Cross Learning for Improved People Counting using Short-Range FMCW Radar

Cited by 24 publications

References 16 publications

Multi-Modal Cross Learning for an FMCW Radar Assisted by Thermal and RGB Cameras to Monitor Gestures and Cooking Processes

Multi-Modal Cross Learning for an FMCW Radar Assisted by Thermal and RGB Cameras to Monitor Gestures and Cooking Processes

Classification of human activity using radar and video multimodal learning

Cross‐Modal Learning

Contact Info

Product

Resources

About