Software defined multi-spectral imaging for Arctic sensor networks

Abstract

Availability of off-the-shelf infrared sensors combined with high definition visible cameras has made possible the construction of a Software Defined Multi-Spectral Imager (SDMSI) combining long-wave, near-infrared and visible imaging. The SDMSI requires a real-time embedded processor to fuse images and to create real-time depth maps for opportunistic uplink in sensor networks. Researchers at Embry Riddle Aeronautical University working with University of Alaska Anchorage at the Arctic Domain Awareness Center and the University of Colorado Boulder have built several versions of a low-cost drop-in-place SDMSI to test alternatives for power efficient image fusion. The SDMSI is intended for use in field applications including marine security, search and rescue operations and environmental surveys in the Arctic region. Based on Arctic marine sensor network mission goals, the team has designed the SDMSI to include features to rank images based on saliency and to provide on camera fusion and depth mapping. A major challenge has been the design of the camera computing system to operate within a 10 to 20 Watt power budget. This paper presents a power analysis of three options: 1) multi-core, 2) field programmable gate array with multi-core, and 3) graphics processing units with multi-core. For each test, power consumed for common fusion workloads has been measured at a range of frame rates and resolutions. Detailed analyses from our power efficiency comparison for workloads specific to stereo depth mapping and sensor fusion are summarized. Preliminary mission feasibility results from testing with off-the-shelf long-wave infrared and visible cameras in Alaska and Arizona are also summarized to demonstrate the value of the SDMSI for applications such as ice tracking, ocean color, soil moisture, animal and marine vessel detection and tracking. The goal is to select the most power efficient solution for the SDMSI for use on UAVs (Unoccupied Aerial Vehicles) and other drop-in-place installations in the Arctic. The prototype selected will be field tested in Alaska in the summer of 2016.

Image and Information Fusion Experiments with a Software-Defined Multi-Spectral Imaging System for Aviation and Marine Sensor Networks

“…Previous work to select the most power efficient processing for the SDMSI has shown that GP-GPU co-processing is one of the most efficient approaches and the NVIDIA Corporation Tegra K1 system on chip was used for all experiments. At peak power, the system draws no more than 20 Watts for processing and for operation of all three cameras concurrently and is nominally operating at 12 Watts of power consumption during bench test measurement with a DMM for the experiments presented [1]. At this time, this is well within requirements for roof operation, but further work on power efficiency is being pursued to enable battery, alternative power source and fuel cell operation of the SDMSI for operation off-grid.…”

“…Image fusion requires spatial registration [3][4][5][7], matching of resolution through pyramidal up-conversion and/or down-conversion at a common aspect ratio and finally blending of pixels if a single fused image is desired rather than side-by-side comparison. Part of the challenge of performing image fusion in real-time is processing and power required, but based on previous work, we have shown this is quite possible for a system operating well below 10 to 20 Watts of total power continuously up to 30 Hz [1]. Furthermore, based on early work, we have determined that this does not require custom hardware [2].…”

“…The system allows for hardware, firmware and software to be open systems, based on embedded Linux running on the processor and emphasis is on the image transforms and salient object detectors that can be run in real-time with power efficiency to support advanced monitoring and observing modes. The power analysis leading to the selection of a GP-GPU co-processor for the SDMSI is presented in detail in previous work [1] and in general the system has a power budget of 20 Watts maximum. The SDMSI system was mounted for testing on the roof of the Embry Riddle Prescott campus and remotely upgraded and accessed with continuous recording of images to compare to intelligent image selection tests.…”

Applications, Evolutions, and Challenges of Drones in Maritime Transport

“…(1) Data fusion: The sensing data acquisition from a single UAV could arguably be insufficient to meet the actual needs of coastal environmental monitoring. A possible solution approach is to combine multiple payloads, such as visible loads, infrared loads, and light detection and ranging (LiDAR), to acquire remote sensing data [139]. By fusing data from different sources with varying characteristics in the same area, the reliability of remote sensing information can be better assured.…”