Lost and Found!

Abstract

We demonstrate an application of finding target persons on a surveillance video. Each visually detected participant is tagged with a smartphone ID and the target person with the query ID is highlighted. This work is motivated by the fact that establishing associations between subjects observed in camera images and messages transmitted from their wireless devices can enable fast and reliable tagging. This is particularly helpful when target pedestrians need to be found on public surveillance footage, without the reliance on facial recognition. The underlying system uses a multimodal approach that leverages WiFi Fine Timing Measurements (FTM) and inertial sensor (IMU) data to associate each visually detected individual with a corresponding smartphone identifier. These smartphone measurements are combined strategically with RGB-D information from the camera, to learn affinity matrices using a multi-modal deep learning network. CCS CONCEPTS• Computing methodologies → Neural networks; • Humancentered computing → Ubiquitous and mobile computing systems and tools.

Layout Sequence Prediction From Noisy Mobile Modality

“…We evaluate our method on two datasets, the Vi-Fi Multi-modal Dataset [18,19] and the H3D Dataset [30], for the problem of layout sequence prediction from noisy mobile data. To assess our proposed method, we require datasets that contain layout sequences (trajectories, bounding boxes, depths, etc.)…”

“…[6]. Vi-Tag is the current state-of-the-art method on the Vi-Fi dataset [18,19] for the vision-motion identity association task, which aims to match identities in cameras in the vision modality with wireless signals in the mobile modality. The core of Vi-Tag is the X-Translator model, which translates the visual modality to the mobile modality and finds the most similar wireless signals to match the identities in both modalities.…”