Road images augmentation with synthetic traffic signs using neural networks

Abstract

Traffic sign recognition is a well-researched problem in computer vision. However, the state of the art methods works only for frequent sign classes, which are well represented in training datasets. We consider the task of rare traffic sign detection and classification. We aim to solve that problem by using synthetic training data. Such training data is obtained by embedding synthetic images of signs in the real photos. We propose three methods for making synthetic signs consistent with a scene in appearance. These methods are based on modern generative adversarial network (GAN) architectures. Our proposed methods allow realistic embedding of rare traffic sign classes that are absent in the training set. We adapt a variational autoencoder for sampling plausible locations of new traffic signs in images. We demonstrate that using a mixture of our synthetic data with real data improves the accuracy of both classifier and detector.

Image Synthesis Pipeline for CNN-Based Sensing Systems

“…For traffic sign detector, we have found that modeling the context in which traffic signs occur is more important than modeling the traffic signs themselves [ 43 ]. We conducted experiments with basic methods for inserting a sign image onto the original image of an unsigned road (the basic methods do not take into account the context of a traffic sign): CGI —samples were obtained by rendering three-dimensional models of traffic signs on pillars in real road images.…”

“…Pasted —In this approach we trained together neural networks for inpainting and processing of embedded traffic sign [ 43 ].…”

“…NN-additional —In this approach we trained a neural network for account object context and thus find better places to insert generated traffic signs [ 43 ].…”

“…The best quality using synthetic samples without neural network placing is achieved on Styled and is equal to 89.13. At the same time, for rare signs AUC (Area Under the Curve) increases from 85.86 to 86.78 [ 43 ] with the hybrid ‘Styled’ approach.…”

Method for creating synthetic data sets for training neural network models for object recognition

“…Теоретически можно генерировать бесконечное количество обучающих изображений с большими вариациями, где разметка осуществляется автоматически. Процесс генерирования данных называют аугментацией [3]. Обучение с искусственными образцами позволяет точно контролировать рендеринг изображений и, следовательно, различные свойства набора данных [4].…”