Fusing two-stream convolutional neural networks for RGB-T object tracking

Li, Chenglong; Wu, Xiaohao; Zhao, Nan; Cao, Xiaochun; Tang, Jin

doi:10.1016/j.neucom.2017.11.068

Cited by 114 publications

(80 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The ranking results are used as patch weights to construct robust RGBT feature representations. To adaptively fuse RGB and thermal infrared modalities, Li et al [20] propose to select most discriminative feature maps in a two-stream convolutional neural network. These methods rely on either handcrafted features or single-adapter deep structures to localize objects, and might be difficult to handle the challenges of significant appearance changes caused by deformation, abrupt motion, background clutter and occlusion, etc.…”

Section: Rgbt Tracking Methodsmentioning

confidence: 99%

Multi-Adapter RGBT Tracking

Zheng

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

Self Cite

128

116

View full text Add to dashboard Cite

The task of RGBT tracking aims to take the complementary advantages from visible spectrum and thermal infrared data to achieve robust visual tracking, and receives more and more attention in recent years. Existing works focus on modality-specific information integration by introducing modality weights to achieve adaptive fusion or learning robust feature representations of different modalities. Although these methods could effectively deploy the modality-specific properties, they ignore the potential values of modality-shared cues as well as instance-aware information, which are crucial for effective fusion of different modalities in RGBT tracking. In this paper, we propose a novel Multi-Adapter convolutional Network (MANet) to jointly perform modality-shared, modality-specific and instance-aware feature learning in an end-to-end trained deep framework for RGBT tracking. We design three kinds of adapters within our network. In a specific, the generality adapter is to extract shared object representations, the modality adapter aims at encoding modality-specific information to deploy their complementary advantages, and the instance adapter is to model the appearance properties and temporal variations of a certain object. Moreover, to reduce computational complexity for real-time demand of visual tracking, we design a parallel structure of generic adapter and modality adapter. Extensive experiments on two RGBT tracking benchmark datasets demonstrate the outstanding performance of the proposed tracker against other state-ofthe-art RGB and RGBT tracking algorithms.

show abstract

Section: Rgbt Tracking Methodsmentioning

confidence: 99%

Multi-Adapter RGBT Tracking

Zheng

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

Self Cite

128

116

View full text Add to dashboard Cite

show abstract

“…The other research stream is to learn robust feature representations via the usage of multimodal data [7], [8], [9]. Li et al [7] propose a weighted sparse representation regularized graph learning approach to construct a graph-based multimodal descriptor, and adopt structured SVM for tracking.…”

Section: A Rgb-thermal Fusion For Trackingmentioning

confidence: 99%

“…Recent studies on RGBT tracking mainly focus on two aspects. The one is aiming to learn robust feature representation via the usage of RGB and thermal data [7], [8], [9] and achieves promising tracking performance. These works rely on either handcrafted features or highly The authors are with School of Computer Science and Technology, Anhui University, Hefei 230601, China.…”

Section: Introductionmentioning

confidence: 99%

Quality-Aware Feature Aggregation Network for Robust RGBT Tracking

Zhu

Tang

et al. 2021

IEEE Trans. Intell. Veh.

Self Cite

115

View full text Add to dashboard Cite

This paper investigates how to perform robust visual tracking in adverse and challenging conditions using complementary visual and thermal infrared data (RGBT tracking). We propose a novel deep network architecture called qualityaware Feature Aggregation Network (FANet) for robust RGBT tracking. Unlike existing RGBT trackers, our FANet aggregates hierarchical deep features within each modality to handle the challenge of significant appearance changes caused by deformation, low illumination, background clutter and occlusion. In particular, we employ the operations of max pooling to transform these hierarchical and multi-resolution features into uniform space with the same resolution, and use 1×1 convolution operation to compress feature dimensions to achieve more effective hierarchical feature aggregation. To model the interactions between RGB and thermal modalities, we elaborately design an adaptive aggregation subnetwork to integrate features from different modalities based on their reliabilities and thus are able to alleviate noise effects introduced by low-quality sources. The whole FANet is trained in an end-to-end manner. Extensive experiments on large-scale benchmark datasets demonstrate the high-accurate performance against other state-of-the-art RGBT tracking methods.

show abstract

“…Although CNN based RGB-T salient object detection algorithms are not well investigated yet, a large number of deep neural networks with RGB-T inputs have been presented for some other computer vision or image processing tasks, such as pedestrian detection [36]- [38], image fusion [50], object tracking [51]- [53]. For example, Wagner et al [37] presented an RGB-T pedestrian detection method by fusing information with CNNs, where information from the RGB and thermal infrared images was integrated via an earlyfusion and a late-fusion based CNN architecture.…”

Section: Rgb-t Salient Object Detectionmentioning

confidence: 99%

RGB-T Salient Object Detection via Fusing Multi-Level CNN Features

Zhang

Huang

Yang

et al. 2020

IEEE Trans. on Image Process.

169

View full text Add to dashboard Cite

RGB-induced salient object detection has recently witnessed substantial progress, which is attributed to the superior feature learning capability of deep convolutional neural networks (CNNs). However, such detections suffer from challenging scenarios characterized by cluttered backgrounds, low-light conditions and variations in illumination. Instead of improving RGB based saliency detection, this paper takes advantage of the complementary benefits of RGB and thermal infrared images. Specifically, we propose a novel end-to-end network for multimodal salient object detection, which turns the challenge of RGB-T saliency detection to a CNN feature fusion problem. To this end, a backbone network (e.g., VGG-16) is first adopted to extract the coarse features from each RGB or thermal infrared image individually, and then several adjacent-depth feature combination (ADFC) modules are designed to extract multi-level refined features for each single-modal input image, considering that features captured at different depths differ in semantic information and visual details. Subsequently, a multi-branch group fusion (MGF) module is employed to capture the crossmodal features by fusing those features from ADFC modules for a RGB-T image pair at each level. Finally, a joint attention guided bi-directional message passing (JABMP) module undertakes the task of saliency prediction via integrating the multi-level fused features from MGF modules. Experimental results on several public RGB-T salient object detection datasets demonstrate the superiorities of our proposed algorithm over the state-of-the-art approaches, especially under challenging conditions, such as poor illumination, complex background and low contrast.

show abstract

Fusing two-stream convolutional neural networks for RGB-T object tracking

Cited by 114 publications

References 10 publications

Multi-Adapter RGBT Tracking

Multi-Adapter RGBT Tracking

Quality-Aware Feature Aggregation Network for Robust RGBT Tracking

RGB-T Salient Object Detection via Fusing Multi-Level CNN Features

Contact Info

Product

Resources

About