Fully Onboard AI-powered Human-Drone Pose Estimation on Ultra-low Power Autonomous Flying Nano-UAVs

Abstract

Artificial intelligence-powered pocket-sized air robots have the potential to revolutionize the Internet-of-Things ecosystem, acting as autonomous, unobtrusive, and ubiquitous smart sensors. With a few cm 2 form-factor, nano-sized unmanned aerial vehicles (UAVs) are the natural befit for indoor humandrone interaction missions, as the pose estimation task we address in this work. However, this scenario is challenged by the nano-UAVs' limited payload and computational power that severely relegates the onboard brain to the sub-100 mW microcontroller unit-class. Our work stands at the intersection of the novel parallel ultra-low-power (PULP) architectural paradigm and our general development methodology for deep neural network (DNN) visual pipelines, i.e., covering from perception to control. Addressing the DNN model design, from training and dataset augmentation to 8-bit quantization and deployment, we demonstrate how a PULP-based processor, aboard a nano-UAV, is sufficient for the real-time execution (up to 135 frame/s) of our novel DNN, called PULP-Frontnet. We showcase how, scaling our model's memory and computational requirement, we can significantly improve the onboard inference (top energy efficiency of 0.43 mJ/frame) with no compromise in the quality-of-result vs. a resource-unconstrained baseline (i.e., full-precision DNN). Field experiments demonstrate a closed-loop top-notch autonomous navigation capability, with a heavily resource-constrained 27grams Crazyflie 2.1 nano-quadrotor. Compared against the control performance achieved using an ideal sensing setup, onboard relative pose inference yields excellent drone behavior in terms of median absolute errors, such as positional (onboard: 41 cm, ideal: 26 cm) and angular (onboard: 3.7°, ideal: 4.1°).

Training Lightweight CNNs for Human-Nanodrone Proximity Interaction from Small Datasets using Background Randomization

“…When autonomous navigation capabilities come to the nano-size class of UAVs, there are three main categories of solutions: i) offloading the computation to some external power-unconstrained base-station [20]; ii) reducing the onboard workload's complexity to minimal functionalities [21]; iii) extending the onboard brain either with general-purpose visual navigation engines based on the ultra-low-power (ULP) heterogeneous model [2,22] or with application-specific processors [23].…”

“…We envision deploying our improved model on a palm-sized UAV that leverages the third solution: the Bitcraze Crazyflie 2.1 2 nano-quadrotor, which is open-source but yet compute/memory-limited robotic platform. To increase the onboard computational energy efficiency, we will leverage the same configuration presented in [2], which extends the basic nano-robot with a pluggable printed circuit board (PCB) called AI-deck. This PCB plays a crucial role in extending the drone's perception capability, with an ULP (i.e., ∼ 4 mW) monochrome Himax HM01B0 camera, able to deliver up to 60 frame/s QVGA images.…”

“…The two models share the same lightweight architecture, which is denoted as 160 × 32 in [2]. This architecture has 303 • 10 3 parameters and can be used for inference onboard the target nano-drone platform at 48 frame/s; it maps a single grayscale 160 × 96 image to the three output pose variables (x, y, φ).…”

“…This can be considered a key perception ability for nano-drones interacting with people. A standard pipeline for solving this problem [1,2] involves: 1) acquiring training sequences in a room equipped with a motion tracking system, while tracking the absolute pose of both the drone and the subject; 2) building a set of training instances, each consisting in a camera frame, and the corresponding ground truth pose of the subject relative to the drone; 3) training a regression model that, given the image, estimates the components of the relative pose.…”

Improving Autonomous Nano-Drones Performance via Automated End-to-End Optimization and Deployment of DNNs

“…In the past years, unmanned aerial vehicles (UAVs) have been adopted in a wide range of applications, such as surveillance and inspection of hazardous areas [1], [2]. Nano-size UAVs, with a form factor of a few centimeters and a weight of tens of grams, are the ideal candidates for fully autonomous indoor navigation as they can safely operate near humans and reach narrow spots with their reduced dimensions [1], [3], [4]. However, these platforms have a total power envelope of a few Watts, of which only 5 − 15% is allotted for computation, making it challenging to deploy real-time navigation pipelines directly onboard [5].…”