Context-Aware Fusion of RGB and Thermal Imagery for Traffic Monitoring

Alldieck, Thiemo; Bahnsen, Chris Holmberg; Moeslund, Thomas B.

doi:10.3390/s16111947

Cited by 33 publications

(12 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Alldieck et al [61] fuse RGB and thermal images from a video stream using a contextual information to access the quality of each image stream to accurately fuse the information from the two sensors. Whereas, methods such as MFNet [62], RTFNet [63], PST900 [64], FuseSeg [65] combine the potential of RGB images along with thermal images using CNN architectures for semantic segmentation of outdoor scenes, providing accurate segmentation results even in presence of degraded lighting conditions.…”

Section: A Scene Understandingmentioning

confidence: 99%

From SLAM to Situational Awareness: Challenges and Survey

Bavle¹,

Sánchez-López²,

Cimarelli³

et al. 2021

Preprint

View full text Add to dashboard Cite

The knowledge that an intelligent and autonomous mobile robot has and is able to acquire of itself and the environment, namely the situation, limits its reasoning, decision-making, and execution skills to efficiently and safely perform complex missions. Situational awareness is a basic capability of humans that has been deeply studied in fields like Psychology, Military, Aerospace, Education, etc., but it has barely been considered in robotics, which has focused on ideas such as sensing, perception, sensor fusion, state estimation, localization and mapping, spatial AI, etc. In our research, we connected the broad multidisciplinary existing knowledge on situational awareness with its counterpart in mobile robotics. In this paper, we survey the state-of-the-art robotics algorithms, we analyze the situational awareness aspects that have been covered by them, and we discuss their missing points. We found out that the existing robotics algorithms are still missing manifold important aspects of situational awareness. As a consequence, we conclude that these missing features are limiting the performance of robotic situational awareness, and further research is needed to overcome this challenge. We see this as an opportunity, and provide our vision for future research on robotic situational awareness.

show abstract

Section: A Scene Understandingmentioning

confidence: 99%

From SLAM to Situational Awareness: Challenges and Survey

Bavle¹,

Sánchez-López²,

Cimarelli³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Multi-Object Tracking (MOT) is used by applications such as ITS, self-driving cars, and traffic-surveillance cameras to detect obstacles and pedestrians at intersections using infrastructure based perception, 144 radar and Light Detection And Ranging (LiDAR) for 3D tracking, 145 video-tracking, 146 or thermal imagery. 147 The goal of MOT is to estimate the trajectories of all objects in a dynamic scene. Different sensors such as video, radar, and LiDAR can be used for MOT.…”

Section: Multi-object Trackingmentioning

confidence: 99%

Deep learning support for intelligent transportation systems

Ibáñez

Contreras-Castillo

Zeadally

2020

Trans Emerging Tel Tech

View full text Add to dashboard Cite

Intelligent Transportation Systems (ITS) help improve the ever-increasing vehicular flow and traffic efficiency in urban traffic to reduce the number of accidents. The generation of massive amounts of data generated by all the digital devices connected to the transportation network enables the creation of datasets to perform an in-depth analysis of the data using deep learning methods. Such methods can help predict traffic performance, automated traffic light management, lane detection, and identifying objects near vehicles to increase the safety and efficiency of ITS. We discuss some of the challenges that need to be solved to achieve seamless integration between ITS and deep learning methods to address issues such as (1) improving traffic flow/transportation logistics, (2) predicting best routes for the transportation of goods, (3) optimal fuel consumption, (4) intelligent environmental conditions perception, (5) traffic speed management, and accident prevention.

show abstract

“…The detection accuracy is shown to be further improved if we allow a context-aware, qualitybased fusion. 19 Context sensitive indicators are introduced 19 to weigh in the soft segmentation performed over individual RGB and infrared data.…”

Section: Related Workmentioning

confidence: 99%

Combining visible and infrared spectrum imagery using machine learning for small unmanned aerial system detection

Goecks

Woods

Valasek

2020

Automatic Target Recognition XXX

View full text Add to dashboard Cite

There is an increasing demand for technology and solutions to counter commercial, off-the-shelf small unmanned aerial systems (sUAS). Advances in machine learning and deep neural networks for object detection, coupled with lower cost and power requirements of cameras, led to promising vision-based solutions for sUAS detection. However, solely relying on the visible spectrum has previously led to reliability issues in low contrast scenarios such as sUAS flying below the treeline and against bright sources of light. Alternatively, due to the relatively high heat signatures emitted from sUAS during flight, a long-wave infrared (LWIR) sensor is able to produce images that clearly contrast the sUAS from its background. However, compared to widely available visible spectrum sensors, LWIR sensors have lower resolution and may produce more false positives when exposed to birds or other heat sources. This research work proposes combining the advantages of the LWIR and visible spectrum sensors using machine learning for vision-based detection of sUAS. Utilizing the heightened background contrast from the LWIR sensor combined and synchronized with the relatively increased resolution of the visible spectrum sensor, a deep learning model was trained to detect the sUAS through previously difficult environments. More specifically, the approach demonstrated effective detection of multiple sUAS flying above and below the treeline, in the presence of heat sources, and glare from the sun. Our approach achieved a detection rate of 71.2 ± 8.3%, improving by 69% when compared to LWIR and by 30.4% when visible spectrum alone, and achieved false alarm rate of 2.7 ± 2.6%, decreasing by 74.1% and by 47.1% when compared to LWIR and visible spectrum alone, respectively, on average, for single and multiple drone scenarios, controlled for the same confidence metric of the machine learning object detector of at least 50%. With a network of these small and affordable sensors, one can accurately estimate the 3D position of the sUAS, which could then be used for elimination or further localization from more narrow sensors, like a fire-control radar (FCR). Videos of the solution's performance can be seen at https://sites.google.com/view/tamudrone-spie2020/.

show abstract

Context-Aware Fusion of RGB and Thermal Imagery for Traffic Monitoring

Cited by 33 publications

References 31 publications

From SLAM to Situational Awareness: Challenges and Survey

From SLAM to Situational Awareness: Challenges and Survey

Deep learning support for intelligent transportation systems

Combining visible and infrared spectrum imagery using machine learning for small unmanned aerial system detection

Contact Info

Product

Resources

About