Mid-fusion of road scene polarization images on pretrained RGB neural networks

Abstract

This work presents a mid-fusion pipeline that can increase the detection performance of a convolutional neural network (RetinaNet) by including polarimetric images even though the network is trained on a large-scale database containing RGB and monochromatic images (Microsoft COCO). Here, the average precision (AP) for each object class quantifies performance. The goal of this work is to evaluate the usefulness of polarimetry for object detection and recognition of road scenes and determine the conditions that will increase AP. Shadows, reflections, albedo, and other object features that reduce RGB image contrast also decrease the AP. This work demonstrates specific cases for which the AP increases using linear Stokes and polarimetric flux images. Images are fused during the neural network evaluation pipeline, which is referred to as mid-fusion. Here, the AP of polarimetric mid-fusion is greater than the RGB AP in 54 out of 80 detection instances. The recall values for cars and buses are similar for RGB and polarimetry, but values increase from 36% to 38% when using polarimetry for detecting people. Videos of linear Stokes images for four different scenes are collected at three different times of the day for two driving directions. Despite this limited dataset and the use of a pretrained network, this work demonstrates selective enhancement of object detection through mid-fusion of polarimetry to neural networks trained on RGB images.

Multispectral polarimetric separability of urban materials

“…The geometric complexity of urban scenes in particular has led some to caution against using passive polarimetry for the purposes of material classification [27] while others merely advise using fewer analytical parameters to avoid overinterpreting for urban materials applications [28]. More recently, various combinations of active and passive polarimetric techniques are being explored especially for improving recognition of correlated shapes and materials for autonomous vehicle navigation in urban situations [29,30,31].…”

Adaptive adjustment with semantic embedding for zero-shot object detection

“…In recent years, object detection tasks combined with deep learning under fully supervised conditions have made significant progress. [1][2][3] Most object-detection methods require large amounts of annotated training data. However, collecting and labeling enough training data is costly in terms of time and human resources.…”

Single-view shape and depth from closed-form polarization models for depolarization-dominated objects

“…[3][4][5] Unfortunately, learning-based techniques are limited by the amount of polarimetric training data currently available. 6 Both physics-based and learning-based approaches typically only make use of the partial polarization information captured by a Stokes camera. Investigating surface normal and absolute depth retrievals from single-view Mueller imaging is an unexplored topic.…”