Frugal following

Abstract

Accurate tracking of objects in the real world is highly desirable in Augmented Reality (AR) to aid proper placement of virtual objects in a user's view. Deep neural networks (DNNs) yield high precision in detecting and tracking objects, but they are energy-heavy and can thus be prohibitive for deployment on mobile devices. Towards reducing energy drain while maintaining good object tracking precision, we develop a novel software framework called MARLIN. MAR-LIN only uses a DNN as needed, to detect new objects or recapture objects that significantly change in appearance. It employs lightweight methods in between DNN executions to track the detected objects with high fidelity. We experiment with several baseline DNN models optimized for mobile devices, and via both offline and live object tracking experiments on two different Android phones (one utilizing a mobile GPU), we show that MARLIN compares favorably in terms of accuracy while saving energy significantly. Specifically, we show that MARLIN reduces the energy consumption by up to 73.3% (compared to an approach that executes the best baseline DNN continuously), and improves accuracy by up to 19× (compared to an approach that infrequently executes the same best baseline DNN). Moreover, while in 75% or more cases, MARLIN incurs at most a 7.36% reduction in location accuracy (using the common IOU metric), in more than 46% of the cases, MARLIN even improves the IOU compared to the continuous, best DNN approach. CCS CONCEPTS • Human-centered computing → Ubiquitous and mobile computing systems and tools.

Energy-Efficient Object Tracking Using Adaptive ROI Subsampling and Deep Reinforcement Learning

“…However, this method is not able to work well with strongly shadowed videos. In [59], the authors propose a software framework titled MARLIN which enables content-driven real-time tracking by switching between deep learning and light-weight techniques. But the method fails in instances when the neural network-based tracker is not triggered in time.…”

Ambient Intelligence for Next-Generation AR

“…By running object detection models on images captured by AR devices, we can detect the type, pose and extents of common objects that are present in real world environments, which provides us with a more in-depth understanding of environmental context and informs the rendering of virtual content. Although current advancements in deep neural networks (DNNs) have shown superior performance in object detection [12,78,99,105], executing large networks on computation-constrained devices such as AR devices and IoT sensors with low latency remains a challenge. To address this, edge-supported architectures are needed to offload computation from the AR devices and IoT sensors and improve the end-to-end latency [70,114,201,205,216].…”

Multi-accelerator Neural Network Inference in Diversely Heterogeneous Embedded Systems

“…This scheduler decides the ODM and accelerator to run such that system objectives are achieved under given constraints. • We deploy a dynamic model loading mechanism that is capable of managing memory resources for each ODM and Feature Related Work Glimpse [2] MARLIN [5] AdaVP [4] RoaD-RuNNer [9] Fast UQ [10] Herald [11] AxoNN [7] SHIFT facilitates switching between ODMs when necessary. • We evaluate the utility and efficiency of SHIFT on three unique off-the-shelf accelerators designed for autonomous systems.…”

“…However, such approaches rely on stable connections to servers, and none of them consider the use of multiple accelerators or multiple DNN models. Marlin [5], and AvaVP [4] are studies which aim to reduce the energy usage onboard a mobile device doing OD. Marlin [5] proposes an approach where, instead of running the DNN every frame, the system alternates between a tracking algorithm and DNN.…”

“…Marlin [5], and AvaVP [4] are studies which aim to reduce the energy usage onboard a mobile device doing OD. Marlin [5] proposes an approach where, instead of running the DNN every frame, the system alternates between a tracking algorithm and DNN. AvaVP [4] extends Marlin [5] by varying the input size of the DNN and skipping frames to adjust the ea-l trade-off during runtime.better latency per frame compared to other onboard-only schemes.…”