Non-Local Aggregation for RGB-D Semantic Segmentation

Zhang, Guodong; Xue, Jing-Hao; Xie, Pengwei; Yang, Sifan; Wang, Guijin

doi:10.1109/lsp.2021.3066071

Cited by 67 publications

(9 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ResNet-101 52.2 77.4 Malleable 2.5D [25] ResNet-101 50.9 76.9 SA-Gate [17] ResNet-101 52.4 77.9 NANet [31] ResNet-101 52.3 77.9 CEN-PSPNet [24] ResNet-152 52.5 77.7 InverseForm [1] ResNet-101 53. our proposed HS3 training scheme, we are able to consistently improve baseline scores as compared to deep supervision. These improvements come with no added inference cost.…”

Section: Results On Cityscapesmentioning

confidence: 99%

HS3: Learning with Proper Task Complexity in Hierarchically Supervised Semantic Segmentation

Borse¹,

Chen²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

While deeply supervised networks are common in recent literature, they typically impose the same learning objective on all transitional layers despite their varying representation powers.In this paper, we propose Hierarchically Supervised Semantic Segmentation (HS3), a training scheme that supervises intermediate layers in a segmentation network to learn meaningful representations by varying task complexity. To enforce a consistent performance vs. complexity trade-off throughout the network, we derive various sets of class clusters to supervise each transitional layer of the network. Furthermore, we devise a fusion framework, HS3-Fuse, to aggregate the hierarchical features generated by these layers. This provides rich semantic contexts and further enhance the final segmentation. Extensive experiments show that our proposed HS3 scheme considerably outperforms deep supervision with no added inference cost. Our proposed HS3-Fuse framework further improves segmentation predictions and achieves state-of-the-art results on two large segmentation benchmarks: NYUD-v2 and Cityscapes.

show abstract

Section: Results On Cityscapesmentioning

confidence: 99%

HS3: Learning with Proper Task Complexity in Hierarchically Supervised Semantic Segmentation

Borse¹,

Chen²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In the second phase, we re-use the weight of pretrianed model to initialize the semantic segmentation network, and train the whole model in an end-to-end manner using Adam optimizer. We employ different initial learning rates of 1×10 −4 , 2×10 Evaluation metrics: We follow previous works [16], [24] to employ PA, MPA, MIoU and FWIoU to evaluate the performance of our static semantic segmentation model. As for dynamic-to-static image translation, we adopt the four popular metrics of L1, L2, PSNR and SSIM to evaluate the quality of the generated static images.…”

Section: A Experimental Settingsmentioning

confidence: 99%

Multi-Modal Visual Place Recognition in Dynamics-Invariant Perception Space

Wu¹,

Wang²,

Sun³

2021

IEEE Signal Process. Lett.

View full text Add to dashboard Cite

Visual place recognition is one of the essential and challenging problems in the fields of robotics. In this letter, we for the first time explore the use of multi-modal fusion of semantic and visual modalities in dynamics-invariant space to improve place recognition in dynamic environments. We achieve this by first designing a novel deep learning architecture to generate the static semantic segmentation and recover the static image directly from the corresponding dynamic image. We then innovatively leverage the spatial-pyramid-matching model to encode the static semantic segmentation into feature vectors. In parallel, the static image is encoded using the popular Bag-ofwords model. On the basis of the above multi-modal features, we finally measure the similarity between the query image and target landmark by the joint similarity of their semantic and visual codes. Extensive experiments demonstrate the effectiveness and robustness of the proposed approach for place recognition in dynamic environments.

show abstract

“…In recent years, multi-modality data have been utilized for semantic segmentation to tackle the issue that the RGB segmentation methods are vulnerable to lighting conditions. And numerous RGB-Depth (RGB-D) [11,12,14,18,20,22,23,32,33] and RGB-T [2,13,15,16,17,19,21,34,35] segmentation algorithms have been proposed. Multi-modality feature fusion is the core challenge for multi-modality semantic segmentation.…”

Section: Multi-modality Semantic Segmentationmentioning

confidence: 99%

“…Generally, feature aggregation is independent of feature extraction [2,11,20,13,17]. In particular, several algorithms [14,22,23] attempt to aggregate the multi-modality features interactively. Namely, they feed the aggregated features into the feature extraction block of the next layer, as depicted by dashed arrows in Figure 1 (b).…”

Section: Introductionmentioning

confidence: 99%

Preface: In honor of the 100th Anniversary of Harbin Institute of Technology

Zhou

2020

Sci. China Technol. Sci.

View full text Add to dashboard Cite

(HIT) was established in 1920 in Harbin, Heilongjiang, China. In 1954, HIT became one of China's first six leading universities. Presently HIT is a member of China's top nine University Union (C9). It is a National Key University with science and engineering as its core and has developed with management, liberal arts, economy, law and other disciplines. This year HIT celebrates its 100th anniversary. HIT's three campuses have graduated more than 300000 students. With strong capabilities in research and innovation, HIT will continue to explore new frontiers and scale greater heights. HIT was the first in China to establish a school of astronautics, the first university to independently develop a small satellite, and the first to realize the scientific experiment of man-machine coordination on in-orbit maintenance (the Tiangong-2 space manipulator). It built the world's first micro-satellite (Longjiang-2) to complete Earth-to-Moon transfer, near-moon braking, and circumlunar flight, and was the first in China to achieve satellite-ground laser link communications. It was a pioneer in realizing target and multi-beam laser auto-alignment of large laser device, as well as space application of magnetic focusing Hall thruster. It developed China's first computer to play chess and talk to humans, China's first new type long range ocean detection radar system, and its first arc-and spot-welding robot. It was the first to reveal the structure of a ligase complex in the HIV virus. It conducted the world's first spaceflight on-orbit demonstration of a flexible solar array system based on shape memory polymer composites. HIT has produced a large number of achievements that have contributed to major space missions, including the first launches of the Long March 7 and the Long March 5, as well as the Tiangong-2 space lab and the Shenzhou-11 manned spacecraft. HIT has signed exchange and cooperation agreements with 278 universities in 39 countries, and since 2011, has led more than 180 international collaborative research projects. In 2011, HIT initiated and jointly established the Association of Sino-Russian Technical Universities (ASRTU) that now counts 67 elite universities as its members. In 2018, HIT became a member of the University of the Arctic, the largest international academic organization in the Circumpolar North. HIT's expansion has seen the growth of its campuses in line with regional development. In the coastal city of Weihai in East China, HIT opened a campus in 1985, focusing on developing marine research and ocean economy, as well as intelligent manufacturing and smart cities. It has undertaken more than 1000 projects, including ocean detection technologies, unmanned vehicles, intelligent robots, new information technology, cyber security, new materials, water purification, and renewable energy, generating hundreds of patents. Seizing the opportunity of the growth of the Guangdong-Hong Kong-Macao Greater Bay Area, HIT's Shenzhen campus is becoming a hub for scientific and technological innovation. Starting from ...

show abstract

Non-Local Aggregation for RGB-D Semantic Segmentation

Cited by 67 publications

References 24 publications

HS3: Learning with Proper Task Complexity in Hierarchically Supervised Semantic Segmentation

HS3: Learning with Proper Task Complexity in Hierarchically Supervised Semantic Segmentation

Multi-Modal Visual Place Recognition in Dynamics-Invariant Perception Space

Preface: In honor of the 100th Anniversary of Harbin Institute of Technology

Contact Info

Product

Resources

About