UnrealStereo: Controlling Hazardous Factors to Analyze Stereo Vision

Abstract

A reliable stereo algorithm is critical for many robotics applications. But textureless and specular regions can easily cause failure by making feature matching difficult. Understanding whether an algorithm is robust to these hazardous regions is important. Although many stereo benchmarks have been developed to evaluate performance, it is hard to quantify the effect of hazardous regions in real images because the location and severity of these regions are unknown. In this paper, we develop a synthetic image generation tool enabling to control hazardous factors, such as making objects more specular or transparent, to produce hazardous regions at different degrees. The densely controlled sampling strategy in virtual worlds enables to effectively stress test stereo algorithms by varying the types and degrees of the hazard. We generate a large synthetic image dataset with automatically computed hazardous regions and analyze algorithms on these regions. The observations from synthetic images are further validated by annotating hazardous regions in real-world datasets Middlebury and KITTI (which gives a sparse sampling of the hazards). Our synthetic image generation tool is based on a game engine Unreal Engine 4 and will be open-source along with the virtual scenes in our experiments. Many publicly available realistic game contents can be used by our tool to provide an enormous resource for development and evaluation of algorithms.

A Survey of Image Synthesis Methods for Visual Machine Learning

“…The aforementioned data sets FlyingThings3D, Monkaa and Driving [MIH*16] provide, apart from optical flow, disparity ground truth maps (Figure 10a). The UnrealStereo data set [ZQC*18], on the other hand, is designed for disparity estimation using non‐procedural and physically based modelled game scenes implemented and rendered in Unreal Engine 4 [UE4]. The majority of synthetic data generation frameworks for depth estimation rely on game/simulator engines for urban and traffic scenes [HUI13, ASS16, GWCV16] (Figure 10b), while Varol et al .…”

Blender as a tool for generating synthetic data

“…Umożliwia to automatyzację procesu generowania i opisywania danych potrzebnych do treningu sieci. Przy użyciu tego typu programów powstało kilka zbiorów danych syntetycznych [12][13][14][15][16][17][18][19][20][21][22] zdjęć i filmów. W większości przypadków, te zbiory danych są kosztowne w wygenerowaniu.…”

A Synthetic Video Dataset for Video Compression Evaluation

“…UE4 is a C++ based tool that is widely used by the games industry and also by movie makers. It has, however, also recently been used for research purposes, for example for the analysis of stereo vision [9] and studying virtual reality (VR) [15]. UE4 has a variety of assets that include models for different scenes and objects.…”

“…foliage, falling leaves), makes capturing an identical scene with different video parameters infeasible. Drawing inspiration from computer vision [8][9][10][11], we propose to address this by the generation of a synthetic video dataset. Such an approach has the benefit of using parameterised models for the production of the synthetic video content.…”

“…The use of synthetic data is a common practise in many research areas, especially in situations where real data may be difficult to acquire, due to budget, time or privacy concerns. For example, in computer vision, synthetic data are used for scene understanding [9] or object recognition [8] and have been proven reliable and useful especially for training neural networks [10,11]. To the best of our knowledge, in the field of video compression, there is only one synthetically-generated dataset [12], designed to simulate a multilens stereoscopic video system with the aim to be used for multiview compression, streaming, or other computer graphics related research.…”