Multi-resolution Cascaded Network with Depth-similar Residual Module for Real-time Semantic Segmentation on RGB-D Images

Zheng, Zhijia; Xie, Donghan; Chen, Chunlin; Zhu, Zhihong

doi:10.1109/icnsc48988.2020.9238079

Cited by 8 publications

(4 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ref. [ 25 ] proposes a depth-similar convolution operation, which converts the unique pixel relationship in the depth map into depth-similar weight dots with the corresponding convolution kernel.…”

Section: Related Workmentioning

confidence: 99%

RGB-D-Based Stair Detection and Estimation Using Deep Learning

Wang

Pei

Qiu

et al. 2023

Sensors

View full text Add to dashboard Cite

Stairs are common vertical traffic structures in buildings, and stair detection tasks are important in environmental perception for autonomous mobile robots. Most existing algorithms have difficulty combining the visual information from binocular sensors effectively and ensuring reliable detection at night and in the case of extremely fuzzy visual clues. To solve these problems, we propose a stair detection network with red-green-blue (RGB) and depth inputs. Specifically, we design a selective module, which can make the network learn the complementary relationship between the RGB feature maps and the depth feature maps and fuse the features effectively in different scenes. In addition, we propose several postprocessing algorithms, including a stair line clustering algorithm and a coordinate transformation algorithm, to obtain the stair geometric parameters. Experiments show that our method has better performance than existing the state-of-the-art deep learning method, and the accuracy, recall, and runtime are improved by 5.64%, 7.97%, and 3.81 ms, respectively. The improved indexes show the effectiveness of the multimodal inputs and the selective module. The estimation values of stair geometric parameters have root mean square errors within 15 mm when ascending stairs and 25 mm when descending stairs. Our method also has extremely fast detection speed, which can meet the requirements of most real-time applications.

show abstract

Section: Related Workmentioning

confidence: 99%

RGB-D-Based Stair Detection and Estimation Using Deep Learning

Wang

Pei

Qiu

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

“…The relevance of multimodal systems is also increasing nowadays. This enables a more comprehensive understanding of the real environment [ 8 ], e.g., for autonomous navigation [ 9 ], medical applications [ 10 , 11 ], for quality control [ 12 ], for interactive robot teaching [ 13 ], for safe robot–human cooperation [ 14 ], or in scene analysis [ 15 , 16 , 17 ]. The projection of transformed points onto a 2D raster image is often used in multimodal image systems for data fusion resp.…”

Section: Introductionmentioning

confidence: 99%

“…The fusion of camera data and modern 3D Light Detection and Ranging (LiDAR) data has many applications, including autonomous and safe control of mobile robots [ 18 ], object detection [ 8 ], and simultaneous localization and mapping (SLAM) [ 24 ]. Depth sensors are also widely used in scene analysis and human–computer interaction, e.g., object detection [ 8 , 17 , 25 ], (semantic) segmentation, [ 15 , 16 , 17 ] or markerless motion capture applications [ 26 ].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Triangle-Mesh-Rasterization-Projection (TMRP): An Algorithm to Project a Point Cloud onto a Consistent, Dense and Accurate 2D Raster Image

Junger

Buch

Notni

2023

Sensors

View full text Add to dashboard Cite

The projection of a point cloud onto a 2D camera image is relevant in the case of various image analysis and enhancement tasks, e.g., (i) in multimodal image processing for data fusion, (ii) in robotic applications and in scene analysis, and (iii) for deep neural networks to generate real datasets with ground truth. The challenges of the current single-shot projection methods, such as simple state-of-the-art projection, conventional, polygon, and deep learning-based upsampling methods or closed source SDK functions of low-cost depth cameras, have been identified. We developed a new way to project point clouds onto a dense, accurate 2D raster image, called Triangle-Mesh-Rasterization-Projection (TMRP). The only gaps that the 2D image still contains with our method are valid gaps that result from the physical limits of the capturing cameras. Dense accuracy is achieved by simultaneously using the 2D neighborhood information (rx,ry) of the 3D coordinates in addition to the points P(X,Y,V). In this way, a fast triangulation interpolation can be performed. The interpolation weights are determined using sub-triangles. Compared to single-shot methods, our algorithm is able to solve the following challenges. This means that: (1) no false gaps or false neighborhoods are generated, (2) the density is XYZ independent, and (3) ambiguities are eliminated. Our TMRP method is also open source, freely available on GitHub, and can be applied to almost any sensor or modality. We also demonstrate the usefulness of our method with four use cases by using the KITTI-2012 dataset or sensors with different modalities. Our goal is to improve recognition tasks and processing optimization in the perception of transparent objects for robotic manufacturing processes.

show abstract

Overview of RGBD semantic segmentation based on deep learning

Zhang

Sheng

et al. 2022

J Ambient Intell Human Comput

View full text Add to dashboard Cite

Multi-resolution Cascaded Network with Depth-similar Residual Module for Real-time Semantic Segmentation on RGB-D Images

Cited by 8 publications

References 22 publications

RGB-D-Based Stair Detection and Estimation Using Deep Learning

RGB-D-Based Stair Detection and Estimation Using Deep Learning

Triangle-Mesh-Rasterization-Projection (TMRP): An Algorithm to Project a Point Cloud onto a Consistent, Dense and Accurate 2D Raster Image

Overview of RGBD semantic segmentation based on deep learning

Contact Info

Product

Resources

About