Semantic Segmentation and Depth Estimation with RGB and DVS Sensor Fusion for Multi-view Driving Perception

Natan, Oskar; Miura, Jun

doi:10.1007/978-3-031-02375-0_26

Cited by 4 publications

(2 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently a plethora of new semantic segmentation methods [16], [17], [18], [19], [20], [21], [22], [23], [24] for visual scene understanding has emerged in the literature, eliciting impressive results. For instance, Nesti et al [19] presented a method that evaluates the robustness of semantic segmentation approaches for autonomous vehicles.…”

Section: A Background and Related Workmentioning

confidence: 99%

Vision-Based Semantic Segmentation in Scene Understanding for Autonomous Driving: Recent Achievements, Challenges, and Outlooks

Muhammad

Hussain

Ullah

et al. 2022

IEEE Trans. Intell. Transport. Syst.

View full text Add to dashboard Cite

Scene understanding plays a crucial role in autonomous driving by utilizing sensory data for contextual information extraction and decision making. Beyond modeling advances, the enabler for vehicles to become aware of their surroundings is the availability of visual sensory data, which expand the vehicular perception and realizes vehicular contextual awareness in real-world environments. Research directions for scene understanding pursued by related studies include person/vehicle detection and segmentation, their transition analysis, lane change, and turns detection, among many others.

show abstract

Section: A Background and Related Workmentioning

confidence: 99%

Vision-Based Semantic Segmentation in Scene Understanding for Autonomous Driving: Recent Achievements, Challenges, and Outlooks

Muhammad

Hussain

Ullah

et al. 2022

IEEE Trans. Intell. Transport. Syst.

View full text Add to dashboard Cite

show abstract

“…1. The perception phase begins with semantic segmentation on RGB image with a standard encoder-decoder network enhanced with several skip connections [29] [30]. The RGB encoder is made of Efficient Net B3 [31] pre-trained on ImageNet [18], hence a normalization process is necessary.…”

Section: A Proposed Modelmentioning

confidence: 99%

DeepIPC: Deeply Integrated Perception and Control for an Autonomous Vehicle in Real Environments

Natan¹,

Miura²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

We propose DeepIPC, an end-to-end multi-task model that handles both perception and control tasks in driving a mobile robot autonomously. The model consists of two main parts, perception and controller modules. The perception module takes RGB image and depth map to perform semantic segmentation and bird's eye view (BEV) semantic mapping along with providing their encoded features. Meanwhile, the controller module processes these features with the measurement of GNSS locations and angular speed to estimate waypoints that come with latent features. Then, two different agents are used to translate waypoints and latent features into a set of navigational controls to drive the robot. The model is evaluated by predicting driving records and performing automated driving under various conditions in the real environment. Based on the experimental results, DeepIPC achieves the best drivability and multi-task performance even with fewer parameters compared to the other models.

show abstract

DeepIPC: Deeply Integrated Perception and Control for an Autonomous Vehicle in Real Environments

Natan,

Miura

2024

IEEE Access

Self Cite

View full text Add to dashboard Cite

In this work, we introduce DeepIPC, a novel end-to-end model tailored for autonomous driving, which seamlessly integrates perception and control tasks. Unlike traditional models that handle these tasks separately, DeepIPC innovatively combines a perception module, which processes RGBD images for semantic segmentation and generates bird's eye view (BEV) mappings, with a controller module that utilizes these insights along with GNSS and angular speed measurements to accurately predict navigational waypoints. This integration allows DeepIPC to efficiently translate complex environmental data into actionable driving commands. Our comprehensive evaluation demonstrates DeepIPC's superior performance in terms of drivability and multi-task efficiency across diverse real-world scenarios, setting a new benchmark for endto-end autonomous driving systems with a leaner model architecture. The experimental results underscore DeepIPC's potential to significantly enhance autonomous vehicular navigation, promising a step forward in the development of autonomous driving technologies. For further insights and replication, we will make our code and datasets available at https://github.com/oskarnatan/DeepIPC.

show abstract

Semantic Segmentation and Depth Estimation with RGB and DVS Sensor Fusion for Multi-view Driving Perception

Cited by 4 publications

References 23 publications

Vision-Based Semantic Segmentation in Scene Understanding for Autonomous Driving: Recent Achievements, Challenges, and Outlooks

Vision-Based Semantic Segmentation in Scene Understanding for Autonomous Driving: Recent Achievements, Challenges, and Outlooks

DeepIPC: Deeply Integrated Perception and Control for an Autonomous Vehicle in Real Environments

DeepIPC: Deeply Integrated Perception and Control for an Autonomous Vehicle in Real Environments

Contact Info

Product

Resources

About