Multitask Multisource Deep Correlation Filter for Remote Sensing Data Fusion

Abstract

With the amount of remote sensing data increasing at an extremely fast pace, machine learning based technique has been shown to perform superiorly in many applications. However, most of existing methods in real-time application are based on single modal image data. Although a few approaches use the different source images to represent the object via fusion scheme, it may not be appropriate for multi-modality information processing. In addition, these methods hardly benefit from the end-to-end network training due to the limitations of implementation difficulty and computational cost. In this paper, we propose a multi-task multi-source information fusion method in the deep learning and correlation filter frameworks, which is applied to the fields of tracking and remote sensing data processing. The contribution of individual layers from different source data inside the deep network model is considered as a task. The proposed method can employ interdependencies among different sources data and tasks to learn deep network parameters and filters jointly to improve the performance. Second, we present an effective object appearance selection scheme to adaptively capture the object appearance changes via effective deep learning network, then integrating information from different modalities to achieve information fusion. Different source information can provide robust performance from different aspects with complementary properties. Third, we further extend the proposed approach to the field of remote sensing for semantic labeling. The layers' sensitivity is utilized to verify the robustness of different classes. Extensively experiments on five benchmarks show that the proposed approach performs favorably against the state-of-the-arts.

A Systematic Literature Review on Multimodal Machine Learning: Applications, Challenges, Gaps and Future Directions

“…[169] has 2 articles in the medical domain. In total, 60 articles were found related to the domain agnostic domain, where, 16 were in classification [170]- [185], 9 were in detection [186]- [194], 11 were in analysis [195]- [205], 8 were in recognition [206]- [213], 9 were in prediction [214]- [222], 1 was in language processing [223], 2 were in image processing [224], [225], 1 was in image retrieval [47], 2 were in integration [226], [227] and 1 was in segmentation [228]. Of 43 articles in the human activity domain, 20 were in recognition [229]- [248], 8 were in detection [249]- [256], 5 were in classification [257]- [261], 4 were in analysis [262]- [265], 3 were in identification [266]- [268] and 1 was in comparison [269], monitoring [270] and assessment [271].…”

“…[98], [100], [104], [116], [125], [127], [187], [206], [241], [326], [365], [395], [411], Image & Numerical [62], [75], [119], [126], [167], [313], [331], [353], [405], [410], Audio & Text & Sensor [384], Audio & Text [180], [282], [377], [391], [392], Text & Signal [109], Text & Numerical [304], [349], Sensor & Signal [240], [242], [258], [389], Sensor & Numerical [183], Signal & Numerical [205], [257], [260], [318]. Figure 10 displays the extracted information related to each modality and data type with the links between them.…”

Hierarchical Markov Random Fields for High Resolution Land Cover Classification of Multisensor and Multiresolution Image Time Series

“…Indeed, the huge amount of data makes the use of Deep Neural Network (DNN) models possible. Many effective multi-task approaches have been recently developed to train DNN models on some large-scale remote-sensing benchmarks (e.g., (Cheng et al 2020, Carvalho et al 2019, Chen et al 2017). The aims of these multi-task methods is to learn an embedding space from different sensors (i.e., task).…”

Visual Object Tracking Based on Modified LeNet-5 and RCCF

“…Due to this visual object tracking, a possibility of achieving multiple difficulties including blur, partial occlusion, camera motion, scale variation, illumination variation, and background clutter. In unconstrained situations, achieving efficient and reliable tracking remains a difficult task [3].…”