Hierarchical multi-modal fusion FCN with attention model for RGB-D tracking

Jiang, Mingxin; Deng, Chao; Shan, Jingsong; Wang, Yuan‐Yuan; Jia, Yongxing; Sun, Xing

doi:10.1016/j.inffus.2018.09.014

Cited by 36 publications

(11 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…I N the field of multi-sensor image fusion, a common task is to combine infrared and visible images. It arises in many applications, such as surveillance [41], object detection and target recognition [16] [15] [11]. The key assumption is that different modalities convey complementary information.…”

Section: Introductionmentioning

confidence: 99%

MDLatLRR: A Novel Decomposition Method for Infrared and Visible Image Fusion

Kittler

2020

IEEE Trans. on Image Process.

364

149

View full text Add to dashboard Cite

Image decomposition is crucial for many image processing tasks, as it allows to extract salient features from source images. A good image decomposition method could lead to a better performance, especially in image fusion tasks. We propose a multi-level image decomposition method based on latent lowrank representation(LatLRR), which is called MDLatLRR. This decomposition method is applicable to many image processing fields. In this paper, we focus on the image fusion task. We build a novel image fusion framework based on MDLatLRR which is used to decompose source images into detail parts(salient features) and base parts. A nuclear-norm based fusion strategy is used to fuse the detail parts and the base parts are fused by an averaging strategy. Compared with other state-of-the-art fusion methods, the proposed algorithm exhibits better fusion performance in both subjective and objective evaluation.

show abstract

Section: Introductionmentioning

confidence: 99%

MDLatLRR: A Novel Decomposition Method for Infrared and Visible Image Fusion

Kittler

2020

IEEE Trans. on Image Process.

364

149

View full text Add to dashboard Cite

show abstract

“…随着深度神经网络在各个领域大展身手 [44] , 再加上相关硬件设施的飞速发展, 科研人员意识到神经网络的强大拟合能力能做到的远不止简单的图像级理解工作, 同时工业界以及应用领域也对计算机视觉信息的理解精度要求逐步提升. 有学者尝试从像素级语义理解的角度解决原有的问题 (如边缘检测 [45] 和目标跟踪 [46,47] ), 此外各方面需求的提升和条件的完善催生了多种新的计算机像素级语义理解任务. 全卷积网络 (fully convolutional networks, FCN) [48] 开创性地抛开了常见神经网络结构中常用的全连接层, 使用端到端 (end to end) 的方式直接使用深度神经网络输出对图像中所有像素的类别预测结果.…”

Section: 在人工智能的应用领域中最基础而常见的图像识别以及分类任务通常被认为是图像级理解任务unclassified

Pixel level semantic understanding: from classification to regression

李学龙¹,

赵致远²

2021

Sci. Sin.-Inf.

View full text Add to dashboard Cite

With the rapid developments in science and technology and the continuous iteration of hardware equipment, artificial intelligence is being widely used in various fields (such as security monitoring, medical assistance, health diagnosis, intelligent recommendation, remote sensing monitoring, and target location). With the gradual expansion of user requirements for intelligent processing tasks, semantic understanding levels and accuracy requirements of intelligent algorithms are also gradually increasing. As such, the pixel-level task of semantic understanding, which requires much higher accuracy than image-level understanding, is attracting increasing attention. Compared to image-level understanding, in addition to the large amount and high precision of pixel-by-pixel output, the internal challenges in pixel-level semantic understanding merit greater attention and research. From the information measurement perspective, and given the unique attributes of pixel-level semantic understanding, in this paper, we present the definition and optimization goal of pixel-level semantic understanding, and derive pixel-level semantic classification and pixel-level semantic regression from the original definition based on the characteristics of actual tasks. We then discuss the degradation and evolution of optimization objectives in these two subcategories, respectively. Based on a detailed investigation, we summarize the development and current status of typical tasks in pixel-level semantic understanding. We consider the difficulties and future development direction of the current pixel-level semantic understanding, perform an in-depth analysis, and suggest feasible solutions to problems that require urgent solutions. Finally, we focus on the opportunities and challenges faced by pixel-level semantic understanding, and even artificial intelligence, in the post-deep-learning era and argue that knowledge guidance and data-driven optimization are fundamental objectives for the future development of artificial intelligence. From the definition and current development of pixel-level semantic understanding, in this work, we extend the current conceptual approach and reflect on the risks faced by the whole field, providing an in-depth discussion and consideration of the developmental direction of related technologies, while clarifying the knowledge base of pixel-level semantic understanding.

show abstract

“…Segmentation has many different prototypes with different architectural designs. To better understand their performance, we compared the most utilized frameworks, including the fully connected network (FCN) [41,42], the pyramid scene parsing network (PSPNet) [43], DeepLab v3+ [10,44], and Mask R-CNN [8]. According to the commonly used website Paper with Code, in which researchers compete for the best performance [45], these algorithms are highly ranked and, thus, are included in this literature review.…”

Section: Semantic Segmentationmentioning

confidence: 99%

An Approach to Semantically Segmenting Building Components and Outdoor Scenes Based on Multichannel Aerial Imagery Datasets

et al. 2021

View full text Add to dashboard Cite

As-is building modeling plays an important role in energy audits and retrofits. However, in order to understand the source(s) of energy loss, researchers must know the semantic information of the buildings and outdoor scenes. Thermal information can potentially be used to distinguish objects that have similar surface colors but are composed of different materials. To utilize both the red–green–blue (RGB) color model and thermal information for the semantic segmentation of buildings and outdoor scenes, we deployed and adapted various pioneering deep convolutional neural network (DCNN) tools that combine RGB information with thermal information to improve the semantic and instance segmentation processes. When both types of information are available, the resulting DCNN models allow us to achieve better segmentation performance. By deploying three case studies, we experimented with our proposed DCNN framework, deploying datasets of building components and outdoor scenes, and testing the models to determine whether the segmentation performance had improved or not. In our observation, the fusion of RGB and thermal information can help the segmentation task in specific cases, but it might also make the neural networks hard to train or deteriorate their prediction performance in some cases. Additionally, different algorithms perform differently in semantic and instance segmentation.

show abstract

Hierarchical multi-modal fusion FCN with attention model for RGB-D tracking

Cited by 36 publications

References 13 publications

MDLatLRR: A Novel Decomposition Method for Infrared and Visible Image Fusion

MDLatLRR: A Novel Decomposition Method for Infrared and Visible Image Fusion

Pixel level semantic understanding: from classification to regression

An Approach to Semantically Segmenting Building Components and Outdoor Scenes Based on Multichannel Aerial Imagery Datasets

Contact Info

Product

Resources

About