Real-Time Instance Segmentation of Traffic Videos for Embedded Devices

Abstract

The paper proposes a novel instance segmentation method for traffic videos devised for deployment on real-time embedded devices. A novel neural network architecture is proposed using a multi-resolution feature extraction backbone and improved network designs for the object detection and instance segmentation branches. A novel post-processing method is introduced to ensure a reduced rate of false detection by evaluating the quality of the output masks. An improved network training procedure is proposed based on a novel label assignment algorithm. An ablation study on speed-vs.-performance trade-off further modifies the two branches and replaces the conventional ResNet-based performance-oriented backbone with a lightweight speed-oriented design. The proposed architectural variations achieve real-time performance when deployed on embedded devices. The experimental results demonstrate that the proposed instance segmentation method for traffic videos outperforms the you only look at coefficients algorithm, the state-of-the-art real-time instance segmentation method. The proposed architecture achieves qualitative results with 31.57 average precision on the COCO dataset, while its speed-oriented variations achieve speeds of up to 66.25 frames per second on the Jetson AGX Xavier module.

Object Detection in Traffic Videos: A Survey

“…Other image-based instance segmentation methods, such as YOLACT [128] and YOLACT++ [129] have also been applied in traffic-related applications [130]. In [131] a novel neural network architecture, namely SOLACT, with a multi-resolution feature extraction backbone is proposed for instance segmentation in traffic surveillance videos with realtime performance on embedded devices.…”

“…Lightweight networks such as SqueezeNet [155], MobileNet [156], and ShuffleNet [157] can be applied to detect objects on edge devices. With the development of smart cities, several studies have attempted to take advantage of the more efficient models to bring advanced AI models to the edge for bandwidth and privacy optimization in traffic surveillance [131], [145], [158], [159]. Highway and urban traffic [165] Obtaining vehicle information Lucking et al [146] 2020 SSD 2D Urban traffic Vehicle counting on edge device Bui et al [142] 2020 YOLOv3 2D 2019 AI City Challenge [41] Vehicle counting Li et al [59] 2020 Faster R-CNN 2D 2020 AI City Challenge [166] Unsupervised anomaly detection Zheng [126] 2020 Mask R-CNN 2D CityFlow dataset [167] Vehicle re-identification Revaud and Humenberger [130] 2021 Mask R-CNN 3D BrnoCompSpeed dataset [160] Speed estimation Zhang [168] 2021 YOLCAT++ 2D Urban intersections (China) Pedestrian and sidewalk detection Chen et al [ The performance of the object detection methods based on handcrafted features could not satisfy the requirements of many real-world applications.…”

“…Transportation departments and transit agencies are interested in computer vision techniques that can process multiple video streams simultaneously without violating the strict real-time requirements. As a result, lightweight object detectors have gained popularity among modern traffic surveillance systems [131], [145], [158], [159].…”

TTIS-YOLO: a traffic target instance segmentation paradigm for complex road scenarios

“…Zhang et al [11] proposed an instance segmentation method based on synthesized images and unlabeled images, considering the correlation between synthesized images, real annotations, and original images from the perspective of experimental data. Panero Martinez et al [12] reduced the false detection rate during the post-processing stage of the instance segmentation model by measuring the quality of the output mask. Zhao et al [13], proposed a spatiotemporal aggregation shuffling attention to address the issue of loss of feature space information, which utilizes the relevant information of frame rates before and after to achieve consistent feature aggregation in the time domain, improving the accuracy of traffic target segmentation.…”

Design of Citrus Fruit Detection System Based on Mobile Platform and Edge Computer Device

“…Fan et al [ 19 ] used UAV low altitude remote sensing technology to obtain crop images with different resolutions, summarized the research and application of crop growth trend, yield estimation, pest, and forage monitoring, and achieved good results; Liu et al [ 20 ] proposed to transform the model based on color space, segment the foreground (crop) and background (soil background) of UAV image, obtain the classified binary image of the image, and then extract the plant number information of corn seedling image by skeleton extraction algorithm. Some studies have deployed target detection algorithms to edge computing devices to complete different tasks [ 21 , 22 , 23 ]. In orchard scenarios, accurate real-time detection or classification goals effectively judge crop health and yield-related management.…”