In-Home Daily-Life Captioning Using Radio Signals

Abstract

This paper aims to caption daily life -i.e., to create a textual description of people's activities and interactions with objects in their homes. Addressing this problem requires novel methods beyond traditional video captioning, as most people would have privacy concerns about deploying cameras throughout their homes. We introduce RF-Diary, a new model for captioning daily life by analyzing the privacy-preserving radio signal in the home with the home's floormap. RF-Diary can further observe and caption people's life through walls and occlusions and in dark settings. In designing RF-Diary, we exploit the ability of radio signals to capture people's 3D dynamics, and use the floormap to help the model learn people's interactions with objects. We also use a multi-modal feature alignment training scheme that leverages existing video-based captioning datasets to improve the performance of our radio-based captioning model. Extensive experimental results demonstrate that RF-Diary generates accurate captions under visible conditions. It also sustains its good performance in dark or occluded settings, where video-based captioning approaches fail to generate meaningful captions. 1

$Radar^2$: Passive Spy Radar Detection and Localization using COTS mmWave Radar

“…In our system, we only focus on the spy radar detection and localization on mmWave band. Meanwhile, there are many other radars working on other frequency bands [9,44]. Actually, our architecture can be easily applied to detect and localize the radars working at other frequency band.…”

“…Novel mmWave radar systems have been explored to detect user's location, activity and vital signs [9,41,43,44] unobtrusively. However, such "unobtrusiveness" of radars can also bring threats to users.…”

A Survey about application based on RF signals

“…Behavior recognition: With the increasing amount of data and the complexity of experimental environment, the limitations of traditional behavior recognition patterns are becoming more and more prominent. In the past few years, the progress of deep learning [26,70,71,72,73,74] has promoted the progress of behavior recognition at an amazing speed. Therefore, researchers use deep learning to extract more advanced features to identify human behavior, and many excellent behavior recognition models based on deep learning have emerged, such as RF action [72], duet [70], Marko [26].…”

“…The first-order logic component obtains the output of identity matching and HMM, and generates the best user behavior explanation. [74] proposed a framework for generating caption description behavior using RF signal, which is composed of four modules. The first module encodes and embeds RF skeleton and floormap together as the feature of the first module.…”

Towards battery-less RF sensing

“…Radio-sensing, the interpretation of time, frequency or phase related features observed at a wireless receiver, for environmental perception has been an active field of research throughout the past decade [1]. In particular, recognition approaches range from interpreting narrow-band timedomain features for simple presence and movement-based predictors [2] to frequency-based approaches exploiting high frequency and wide bandwidth for the recognition of minute movement [3]. In recent years, radar-based mechanisms have become popular [4] and we are witnessing the integration of radar-based sensing technology into off-the-shelf commercial devices [5].…”