Tracking in Aerial Hyperspectral Videos Using Deep Kernelized Correlation Filters

Abstract

Hyperspectral imaging holds enormous potential to improve the state-of-the-art in aerial vehicle tracking with low spatial and temporal resolutions. Recently, adaptive multi-modal hyperspectral sensors have attracted growing interest due to their ability to record extended data quickly from aerial platforms. In this study, we apply popular concepts from traditional object tracking, namely (1) Kernelized Correlation Filters (KCF) and (2) Deep Convolutional Neural Network (CNN) features to aerial tracking in hyperspectral domain. We propose the Deep Hyperspectral Kernelized Correlation Filter based tracker (DeepHKCF) to efficiently track aerial vehicles using an adaptive multi-modal hyperspectral sensor. We address low temporal resolution by designing a single KCF-in-multiple Regions-of-Interest (ROIs) approach to cover a reasonably large area. To increase the speed of deep convolutional features extraction from multiple ROIs, we design an effective ROI mapping strategy. The proposed tracker also provides flexibility to couple with the more advanced correlation filter trackers. The DeepHKCF tracker performs exceptionally well with deep features set up in a synthetic hyperspectral video generated by the Digital Imaging and Remote Sensing Image Generation (DIRSIG) software. Additionally, we generate a large, synthetic, single-channel dataset using DIRSIG to perform vehicle classification in the Wide Area Motion Imagery (WAMI) platform. This way, the high-fidelity of the DIRSIG software is proved and a large scale aerial vehicle classification dataset is released to support studies on vehicle detection and tracking in the WAMI platform.

Error Bounded Foreground and Background Modeling for Moving Object Detection in Satellite Videos

“…S ATELLITE videos, provided by JiLin-1 [1] and Skybox [2], have become available since a few years ago. The dense temporal information contained in these videos facilitates the solution to various surveillance problems, such as Moving Object Detection (MOD) [3], [4] and target tracking [5], [6], [7], [8]. The results will be valuable for new applications, including traffic monitoring, analysis and control at a large scale from space [9].…”

Material Based Object Tracking in Hyperspectral Videos

“…In this experiment, we compare the effectiveness of five feature extractors, including spectrum, histogram of oriented gradients (HOG), band-wise HOG (BHOG) [29], abundances, SSHMG, and SSHMG combined with material abundances (abbreviated as MHT). For spectrum, the raw spectral response at each pixel was employed as the feature for tracking.…”

“…Pixel-wise spectral reflectance was adopted as the feature for object tracking in early attempts [25]- [28], but arXiv:1812.04179v5 [cs.CV] 10 Jul 2019 the spatial structure is ignored. Alternatively, Uzkent et al [29] proposed a deep kernelized correlation filter based method (DeepHKCF) for aerial object tracking at the sacrifice of valuable spectral information, in which an HSI was converted to false-color image before passing to a deep convolutional neural network. Qian et al [30] selected a set of patches as convolutional kernels for each band to extract features, but the correlations among bands were neglected.…”

AeroRIT: A New Scene for Hyperspectral Image Analysis

“…The pixels were labeled with ENVI 3 , using individual hyperspectral signatures and the geo-registered RGB images as references. As the RGB images do not form a continuous flight line (framing camera 3 data analyses were done using ENVI version 4.8.2 (Exelis Visual Information Solutions, Boulder, Colorado). pattern) and are more in short burst captures format, we only labeled the hyperspectral scene and use it in our analysis.…”

“…There has been interest in using CNNs for analyzing remote sensing imagery [3], [4], [5], [6], [7]. Uzkent et al adapted correlation filters trained on RGB images with HSI bands to successfully track cars in moving platform scenarios [3]. Hughes et al used a siamese CNN architecture to match high resolution optical images with their corresponding synthetic aperture radar images [4].…”