RF-based 3D skeletons

Zhao, M.; Tian, Yonglong; Zhao, Hang; Alsheikh, Mohammad Abu; Li, Tianhong; Hristov, Rumen; Kabelac, Zachary; Katabi, Dina; Torralba, Antonio

doi:10.1145/3230543.3230579

Cited by 299 publications

(162 citation statements)

References 26 publications

Supporting

Mentioning

147

Contrasting

Order By: Relevance

“…In the rest of this section, we will describe how we transform wireless signals to 3D skeleton sequences, and how we infer actions from such skeleton sequences -i.e., the yellow and green boxes in Figure 3. Transforming visual data from a multi-camera system to 3D skeletons can be done by extracting 2D skeletons from images using an algorithm like AlphaPose and then triangulating the 2D keypoints to generate 3D skeletons, as commonly done in the literature [15,45].…”

Section: Methodsmentioning

confidence: 99%

“…We use a type of radio commonly used in past work on RF-based action recognition [45,24,41,7,28,33,16,42,46,44]. The radio generates a waveform called FMCW and operates between 5.4 and 7.2 GHz.…”

Section: Radio Frequency Signals Primermentioning

confidence: 99%

“…To generate human skeletons from wireless signals, we adopt the architecture from [45]. Specifically, the skeleton generation network (the orange box in Figure 3) takes in wireless signals in the form of horizontal and vertical heatmaps shown in Figure 2, and generates multi-person 3D skeletons.…”

Section: Skeleton-generation From Wireless Signalsmentioning

confidence: 99%

See 2 more Smart Citations

Making the Invisible Visible: Action Recognition Through Walls and Occlusions

Li¹,

Fan²,

Zhao³

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

111

View full text Add to dashboard Cite

Understanding people's actions and interactions typically depends on seeing them. Automating the process of action recognition from visual data has been the topic of much research in the computer vision community. But what if it is too dark, or if the person is occluded or behind a wall? In this paper, we introduce a neural network model that can detect human actions through walls and occlusions, and in poor lighting conditions. Our model takes radio frequency (RF) signals as input, generates 3D human skeletons as an intermediate representation, and recognizes actions and interactions of multiple people over time. By translating the input to an intermediate skeleton-based representation, our model can learn from both vision-based and RF-based datasets, and allow the two tasks to help each other. We show that our model achieves comparable accuracy to vision-based action recognition systems in visible scenarios, yet continues to work accurately when people are not visible, hence addressing scenarios that are beyond the limit of today's vision-based action recognition.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Radio Frequency Signals Primermentioning

confidence: 99%

See 1 more Smart Citation

Making the Invisible Visible: Action Recognition Through Walls and Occlusions

Li¹,

Fan²,

Zhao³

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

111

View full text Add to dashboard Cite

show abstract

“…Their models are trained on the server side, and inferences are performed locally on mobile devices. More recently, Zhao et al design a 4D CNN framework (3D for the spatial dimension + 1D for the temporal dimension) to reconstruct human skeletons using radio frequency signals [287]. This novel approach resembles virtual "X-ray", enabling to accurately estimate human poses, without requiring an actual camera.…”

Section: Mobilementioning

confidence: 99%

Deep Learning in Mobile and Wireless Networking: A Survey

Zhang

Patras

Haddadi

2019

IEEE Commun. Surv. Tutorials

1,335

838

View full text Add to dashboard Cite

The rapid uptake of mobile devices and the rising popularity of mobile applications and services pose unprecedented demands on mobile and wireless networking infrastructure. Upcoming 5G systems are evolving to support exploding mobile traffic volumes, real-time extraction of fine-grained analytics, and agile management of network resources, so as to maximize user experience. Fulfilling these tasks is challenging, as mobile environments are increasingly complex, heterogeneous, and evolving. One potential solution is to resort to advanced machine learning techniques, in order to help manage the rise in data volumes and algorithm-driven applications. The recent success of deep learning underpins new and powerful tools that tackle problems in this space.In this paper we bridge the gap between deep learning and mobile and wireless networking research, by presenting a comprehensive survey of the crossovers between the two areas. We first briefly introduce essential background and state-of-theart in deep learning techniques with potential applications to networking. We then discuss several techniques and platforms that facilitate the efficient deployment of deep learning onto mobile systems. Subsequently, we provide an encyclopedic review of mobile and wireless networking research based on deep learning, which we categorize by different domains. Drawing from our experience, we discuss how to tailor deep learning to mobile environments. We complete this survey by pinpointing current challenges and open future directions for research.

show abstract

“…With the subtle signal variations caused by the target, the target's rich information can be sensed without any device attached to the target. Contactless wireless sensing has enabled a large variety of exciting new applications including indoor localization, 3,4 activity/gesture recognition, 5,6 fall detection, 7 respiration monitoring, 8 emotion sensing, 9 material identification, 10,11 room layout mapping, 12 imaging, 13 etc. Some of the applications enabled by ubiquitous commodity WiFi hardware are shown in Figure 1.…”

Section: Introductionmentioning

confidence: 99%

Sensing Our World Using Wireless Signals

Chen¹,

Chen²,

Feng³

et al. 2019

IEEE Internet Comput.

View full text Add to dashboard Cite

In the last few years, pervasive wireless signals have been established as a powerful medium for ubiquitous sensing. Wireless sensing quickly becomes an active cross-disciplinary research area which involves wireless communication, signal processing, human-computer interaction, machine learning and even hardware design. This article provides an accessible introduction to this exciting area and presents the key technologies enabling it. It highlights the latest achievements in this field, and the unaddressed research challenges as well as how emerging wireless standards can better support this promising technique.

show abstract

RF-based 3D skeletons

Cited by 299 publications

References 26 publications

Making the Invisible Visible: Action Recognition Through Walls and Occlusions

Making the Invisible Visible: Action Recognition Through Walls and Occlusions

Deep Learning in Mobile and Wireless Networking: A Survey

Sensing Our World Using Wireless Signals

Contact Info

Product

Resources

About