A multimodal hyper-fusion transformer for remote sensing image classification

Ma, Mengru; Ma, Wanhong; Jiao, Licheng; Liu, Xu; Feng, Zhixi; Liu, Fang; Yang, Shuyuan

doi:10.1016/j.inffus.2023.03.005

Cited by 19 publications

(3 citation statements)

References 103 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Experimental results show that our method improves the SCL by 6.5% and achieves superior performance compared with recent works. For example, it outperforms a recent contrastive learning based method TSC (Li et al 2021) by 5.3% on ImageNet-LT. Our method can be flexibly combined with ensemble-based methods like RIDE (Wang et al 2020), which achieves the overall accuracy of 74.9% on the iNaturaList 2018, outperforming the recent work CR (Ma et al 2023) by 1.4% in overall accuracy.…”

Section: Introductionmentioning

confidence: 91%

Decoupled Contrastive Learning for Long-Tailed Recognition

Xuan,

Zhang

2024

AAAI

View full text Add to dashboard Cite

Supervised Contrastive Loss (SCL) is popular in visual representation learning. Given an anchor image, SCL pulls two types of positive samples, i.e., its augmentation and other images from the same class together, while pushes negative images apart to optimize the learned embedding. In the scenario of long-tailed recognition, where the number of samples in each class is imbalanced, treating two types of positive samples equally leads to the biased optimization for intra-category distance. In addition, similarity relationship among negative samples, that are ignored by SCL, also presents meaningful semantic cues. To improve the performance on long-tailed recognition, this paper addresses those two issues of SCL by decoupling the training objective. Specifically, it decouples two types of positives in SCL and optimizes their relations toward different objectives to alleviate the influence of the imbalanced dataset. We further propose a patch-based self distillation to transfer knowledge from head to tail classes to relieve the under-representation of tail classes. It uses patch-based features to mine shared visual patterns among different instances and leverages a self distillation procedure to transfer such knowledge. Experiments on different long-tailed classification benchmarks demonstrate the superiority of our method. For instance, it achieves the 57.7% top-1 accuracy on the ImageNet-LT dataset. Combined with the ensemble-based method, the performance can be further boosted to 59.7%, which substantially outperforms many recent works. Our code will be released.

show abstract

Section: Introductionmentioning

confidence: 91%

Decoupled Contrastive Learning for Long-Tailed Recognition

Xuan,

Zhang

2024

AAAI

View full text Add to dashboard Cite

show abstract

“…The key challenge of person re-ID lies in the style disparities between cameras due to different camera settings, viewpoints, backgrounds, illuminations and resolutions [27,28]. Recently, thanks to the development of advanced network structure and deep learning [29][30][31][32], person re-ID has achieved remarkable progress. Zhu et al [10] addressed the challenge of viewpoint variation by projecting the features of people with different viewpoints into a unified space and modeling the representations on identityand viewpoint-level.…”

Section: Related Work 21 Deep Learning Person Re-identificationmentioning

confidence: 99%

Multi-Scale Image- and Feature-Level Alignment for Cross-Resolution Person Re-Identification

Zhang,

Wang,

Zhang

et al. 2024

Remote Sensing

View full text Add to dashboard Cite

Cross-Resolution Person Re-Identification (re-ID) aims to match images with disparate resolutions arising from variations in camera hardware and shooting distances. Most conventional works utilize Super-Resolution (SR) models to recover Low Resolution (LR) images to High Resolution (HR) images. However, because the SR models cannot completely compensate for the missing information in the LR images, there is still a large gap between the HR image recovered from the LR images and the real HR images. To tackle this challenge, we propose a novel Multi-Scale Image- and Feature-Level Alignment (MSIFLA) framework to align the images on multiple resolution scales at both the image and feature level. Specifically, (i) we design a Cascaded Multi-Scale Resolution Reconstruction (CMSR2) module, which is composed of three cascaded Image Reconstruction (IR) networks, and can continuously reconstruct multiple variables of different resolution scales from low to high for each image, regardless of image resolution. The reconstructed images with specific resolution scales are of similar distribution; therefore, the images are aligned on multiple resolution scales at the image level. (ii) We propose a Multi-Resolution Representation Learning (MR2L) module which consists of three-person re-ID networks to encourage the IR models to preserve the ID-discriminative information during training separately. Each re-ID network focuses on mining discriminative information from a specific scale without the disturbance from various resolutions. By matching the extracted features on three resolution scales, the images with different resolutions are also aligned at the feature-level. We conduct extensive experiments on multiple public cross-resolution person re-ID datasets to demonstrate the superiority of the proposed method. In addition, the generalization of MSIFLA in handling cross-resolution retrieval tasks is verified on the UAV vehicle dataset.

show abstract

“…Bachmann et al [37] introduced Constant Curvature Graph Convolutional Networks (CC-GCNs) that can be applied to node classification and distortion minimization tasks in non-Euclidean geometries. Additionally, Ma et al [38] proposed a curvature regularization approach to address the issue of model bias caused by curvature imbalance in deep neural networks. Other researchers, such as Lin et al [39] and Arvanitidis et al [40], examined the curvature of deep generative models and developed new architectures and approaches to improve their performance.…”

Section: Curvature Of Feature Space Geometrymentioning

confidence: 99%

The Geometry of Feature Space in Deep Learning Models: A Holistic Perspective and Comprehensive Review

Lee

2023

Mathematics

View full text Add to dashboard Cite

As the field of deep learning experiences a meteoric rise, the urgency to decipher the complex geometric properties of feature spaces, which underlie the effectiveness of diverse learning algorithms and optimization techniques, has become paramount. In this scholarly review, a comprehensive, holistic outlook on the geometry of feature spaces in deep learning models is provided in order to thoroughly probe the interconnections between feature spaces and a multitude of influential factors such as activation functions, normalization methods, and model architectures. The exploration commences with an all-encompassing examination of deep learning models, followed by a rigorous dissection of feature space geometry, delving into manifold structures, curvature, wide neural networks and Gaussian processes, critical points and loss landscapes, singular value spectra, and adversarial robustness, among other notable topics. Moreover, transfer learning and disentangled representations in feature space are illuminated, accentuating the progress and challenges in these areas. In conclusion, the challenges and future research directions in the domain of feature space geometry are outlined, emphasizing the significance of comprehending overparameterized models, unsupervised and semi-supervised learning, interpretable feature space geometry, topological analysis, and multimodal and multi-task learning. Embracing a holistic perspective, this review aspires to serve as an exhaustive guide for researchers and practitioners alike, clarifying the intricacies of the geometry of feature spaces in deep learning models and mapping the trajectory for future advancements in this enigmatic and enthralling domain.

show abstract

A multimodal hyper-fusion transformer for remote sensing image classification

Cited by 19 publications

References 103 publications

Decoupled Contrastive Learning for Long-Tailed Recognition

Decoupled Contrastive Learning for Long-Tailed Recognition

Multi-Scale Image- and Feature-Level Alignment for Cross-Resolution Person Re-Identification

The Geometry of Feature Space in Deep Learning Models: A Holistic Perspective and Comprehensive Review

Contact Info

Product

Resources

About