Synthetic data for visual machine learning : A data-centric approach

Abstract

Deep learning allows computers to learn from observations, or else training data. Successful application development requires skills in neural network design, adequate computational resources, and a training data distribution that covers the application domain. We are currently witnessing an artificial intelligence (AI) outbreak with enough computational power to train very deep networks and build models that achieve similar or better than human performance. The crucial factor for the algorithms to succeed has proven to be the training data fed to the learning process. Too little or low quality or out-of-the-target distribution data will lead to poorly performing models no matter the capacity and the data regularization methods.This thesis is a data-centric approach to AI and presents a set of contributions related to synthesizing images for training supervised visual machine learning. It is motivated by the profound potential of synthetic data in cases of low availability of captured data, expensive acquisition and annotation, and privacy and ethical issues. The presented work aims to generate images similar to samples drawn from the target distribution and evaluate the generated data as the sole training data source and in conjunction with captured imagery. For this, two synthesis methods are explored: computer graphics and generative modeling. Computer graphics-based generation methods and synthetic datasets for computer vision tasks are thoroughly reviewed. In the same context, a system employing procedural modeling and physically-based rendering is introduced for data generation for urban scene understanding. The scheme is flexible, easily scalable, and produces complex and diverse images with pixel-perfect annotations at no cost. Generative Adversarial Networks (GANs) are also used to generate images for small data scenarios augmentation. The strategy advances the model's performance and robustness. Finally, ensembles of independently trained GANs investigate ways to improve images' diversity and create synthetic data to serve as the only training source.The application areas of the presented contributions relate to two image modalities, natural and histopathology images, to cover different aspects in the generation methods and the tasks' characteristics and requirements. There are showcased synthesized examples of natural images for automotive applications and weather classification, and histopathology images for breast cancer and colon adenocarcinoma metastasis detection. This thesis, as a whole, promotes data-centric supervised deep learning development by highlighting the potential of synthetic data as a training data resource. It emphasizes the control over the formation process, the ability of multi-modality formats, and the automatic generation of annotations.

Leveraging Synthetic Data to Learn Video Stabilization Under Adverse Conditions

“…The use of synthetic data in the computer vision community has shown to be a promising solution to overcome the lack of suitable data for training supervised learning models [25,26,6,4,5,7]. Adapting a specific video game to generate synthetic data with its corresponding ground-truth for the task of semantic segmentation was presented by Richter et al [26], who modified the game Grand Theft Auto V for that purpose.…”

“…Synthetic data have shown a great progress in the field of computer vision [4,5,6,7]. Its increase in popularity started to attract many researchers to apply it for different computer vision problems.…”

Automatic Image Generation Pipeline for Instance Segmentation of Deformable Linear Objects

“…Deep learning requires large amounts of labeled training data, hence the lack of data is a current and significant challenge [ 1 ]. Large image datasets are available, especially for autonomous driving applications and household scenarios, but manual real-world image acquisition and labeling of new datasets involve considerable time and effort [ 2 ]. In particular, annotating segmentation masks is laborious and requires much more time than labeling bounding boxes.…”

“…In particular, annotating segmentation masks is laborious and requires much more time than labeling bounding boxes. Other challenges, when using real-world training data, can be inconsistent label quality and underrepresentation of rare events in the dataset [ 2 ]. Synthetic image data generation and an integrated computation of associated labels can compensate for these challenges.…”