RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving

El, Madawy, Khaled; Rashed, Hazem; Sallab, Ahmad El; Nasr, Omar A.; Kamel, Hanan; Yogamani, Senthil

doi:10.48550/arxiv.1906.00208

Cited by 4 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(2) Ant species that aggressively trim and/or consume acacia leaves (i.e. C. nigriceps and T. penzigi) make their host trees less verdant, which would likely be detected with infrared or RGB colour sensors (Madawy et al, 2019). (3) Tetraponera penzigi is the best disperser among the four ant species (Stanton et al, 2002).…”

Section: Limitations and Future Improvementsmentioning

confidence: 99%

Automated detection of an insect‐induced keystone vegetation phenotype using airborne LiDAR

Wang,

Huben,

Boucher

et al. 2024

Methods Ecol Evol

View full text Add to dashboard Cite

Ecologists, foresters and conservation practitioners need ‘biodiversity scanners’ to effectively inventory biodiversity, audit conservation progress and track changes in ecosystem function. Quantifying biological diversity using remote sensing methods remains challenging, especially for small invertebrates. However, insect aggregations can drastically alter landscapes and vegetation, and these ‘extended phenotypes’ could serve as environmental landmarks of insect presence in remotely sensed data. To test the feasibility of this approach, we studied symbiotic ants that alter the canopy shape of whistling thorn acacias (Acacia [syn. Vachellia] drepanolobium), a keystone tree species of the black cotton soils of east African savannas. We demonstrate a protocol for using light detection and ranging (LiDAR) data to collect, prepare (including a customizable tree‐segmentation algorithm) and apply a convolutional neural network‐based classification for the detection of ant‐inhabited acacia tree phenotypic variations. Applying this protocol enabled us to effectively detect intra‐specific tree phenotypic variation induced by insects. Surveying ant occupancy across 16 ha and 9680 acacia trees took 1000 work hours, whereas surveyed patterns of ant distribution were replicated by our trained classifier using only an hour‐long airborne LiDAR collection time. We suggest that large‐scale surveys of insect occupancy (including insect‐vectored disease) can be automated through a combination of airborne LiDAR and machine learning.

show abstract

Section: Limitations and Future Improvementsmentioning

confidence: 99%

Automated detection of an insect‐induced keystone vegetation phenotype using airborne LiDAR

Wang,

Huben,

Boucher

et al. 2024

Methods Ecol Evol

View full text Add to dashboard Cite

show abstract

“…Another approach [21] used a network consisting of three CNN encoder-decoder sub-networks to fuse RGB images, lidar, and radar for road detection. Additionally, Khaled et al in [23] use two networks, SqueezeSeg and PointSeg, for semantic segmentation and to apply a feature fusion level to fuse a 3D lidar with an RGB camera for pedestrian detection.…”

Section: Introductionmentioning

confidence: 99%

“…Semantic segmentation is a well-developed algorithm based on image data [23] and that leverages recent work on sensor fusion carried out in [21] to handle multi-modal fusion. In this work, an asymmetrical CNN architecture was used that consisted of three encoder sub-networks, where each sub-network was assigned to a particular sensor stream: camera, lidar, and radar.…”

Section: Introductionmentioning

confidence: 99%

A Preliminary Study of Deep Learning Sensor Fusion for Pedestrian Detection

Plascencia

García-Gómez²,

Perez

et al. 2023

Sensors

View full text Add to dashboard Cite

Most pedestrian detection methods focus on bounding boxes based on fusing RGB with lidar. These methods do not relate to how the human eye perceives objects in the real world. Furthermore, lidar and vision can have difficulty detecting pedestrians in scattered environments, and radar can be used to overcome this problem. Therefore, the motivation of this work is to explore, as a preliminary step, the feasibility of fusing lidar, radar, and RGB for pedestrian detection that potentially can be used for autonomous driving that uses a fully connected convolutional neural network architecture for multimodal sensors. The core of the network is based on SegNet, a pixel-wise semantic segmentation network. In this context, lidar and radar were incorporated by transforming them from 3D pointclouds into 2D gray images with 16-bit depths, and RGB images were incorporated with three channels. The proposed architecture uses a single SegNet for each sensor reading, and the outputs are then applied to a fully connected neural network to fuse the three modalities of sensors. Afterwards, an up-sampling network is applied to recover the fused data. Additionally, a custom dataset of 60 images was proposed for training the architecture, with an additional 10 for evaluation and 10 for testing, giving a total of 80 images. The experiment results show a training mean pixel accuracy of 99.7% and a training mean intersection over union of 99.5%. Also, the testing mean of the IoU was 94.4%, and the testing pixel accuracy was 96.2%. These metric results have successfully demonstrated the effectiveness of using semantic segmentation for pedestrian detection under the modalities of three sensors. Despite some overfitting in the model during experimentation, it performed well in detecting people in test mode. Therefore, it is worth emphasizing that the focus of this work is to show that this method is feasible to be used, as it works regardless of the size of the dataset. Also, a bigger dataset would be necessary to achieve a more appropiate training. This method gives the advantage of detecting pedestrians as the human eye does, thereby resulting in less ambiguity. Additionally, this work has also proposed an extrinsic calibration matrix method for sensor alignment between radar and lidar based on singular value decomposition.

show abstract

“…Panoramic camera and multi-sensor fusion are good solutions [6] [7]. For example, by installing multiple cameras on the vehicle, or adding additional ultrasonic radar and LiDAR sensors, we can increase the amount of acquired information [8] [9]. However, these methods require additional calibration and matching of multiple sensors, and repeated calibration and matching are required for different designed hardware collocation methods [10].…”

Section: Introductionmentioning

confidence: 99%

Universal Semantic Segmentation for Fisheye Urban Driving Images

Yang

Xiang

et al. 2020

2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC)

View full text Add to dashboard Cite

Semantic segmentation is a critical method in the field of autonomous driving. When performing semantic image segmentation, a wider field of view (FoV) helps to obtain more information about the surrounding environment, making automatic driving safer and more reliable, which could be offered by fisheye cameras. However, large public fisheye data sets are not available, and the fisheye images captured by the fisheye camera with large FoV comes with large distortion, so commonly-used semantic segmentation model cannot be directly utilized. In this paper, a seven degrees of freedom (DoF) augmentation method is proposed to transform rectilinear image to fisheye image in a more comprehensive way. In the training process, rectilinear images are transformed into fisheye images in seven DoF, which simulates the fisheye images taken by cameras of different positions, orientations and focal lengths. The result shows that training with the seven-DoF augmentation can evidently improve the model's accuracy and robustness against different distorted fisheye data. This seven-DoF augmentation provides an universal semantic segmentation solution for fisheye cameras in different autonomous driving applications. Also, we provide specific parameter settings of the augmentation for autonomous driving. At last, we tested our universal semantic segmentation model on real fisheye images and obtained satisfactory results. The code and configurations are released at https://github.com/Yaozhuwa/FisheyeSeg.

show abstract

RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving

Cited by 4 publications

References 0 publications

Automated detection of an insect‐induced keystone vegetation phenotype using airborne LiDAR

Automated detection of an insect‐induced keystone vegetation phenotype using airborne LiDAR

A Preliminary Study of Deep Learning Sensor Fusion for Pedestrian Detection

Universal Semantic Segmentation for Fisheye Urban Driving Images

Contact Info

Product

Resources

About