Sim4CV: A Photo-Realistic Simulator for Computer Vision Applications

Müller, Matthias; Casser, Vincent; Lahoud, Jean; Smith, Neil; Ghanem, Bernard

doi:10.1007/s11263-018-1073-7

Cited by 142 publications

(92 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Experimental Setup. We use our UAV racing environment in Sim4CV [16] (see Figure 1); we design seven racing tracks for training and seven tracks for testing. To avoid user bias, we collect online images and trace their contours to create uniquely stylized tracks.…”

Section: Methodsmentioning

confidence: 99%

“…Here we provide additional results and the recorded paths of the UAV networks and human pilots evaluations in the paper. All results were recorded as logs during testing inside Sim4CV [16] allowing plotting on the GUI track interface developed for the paper. The logs record stick input, position, orientation and velocity.…”

Section: A Supplementary Materialsmentioning

confidence: 99%

“…We use Sim4CV [16] to simulate UAV racing with accurate physics using the Unreal Engine 4 (UE4). [17].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Learning a Controller Fusion Network by Online Trajectory Filtering for Vision-Based UAV Racing

Müller

Casser

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

Self Cite

View full text Add to dashboard Cite

Autonomous UAV racing has recently emerged as an interesting research problem. The dream is to beat humans in this new fast-paced sport. A common approach is to learn an end-to-end policy that directly predicts controls from raw images by imitating an expert. However, such a policy is limited by the expert it imitates and scaling to other environments and vehicle dynamics is difficult. One approach to overcome the drawbacks of an end-to-end policy is to train a network only on the perception task and handle control with a PID or MPC controller. However, a single controller must be extensively tuned and cannot usually cover the whole state space. In this paper, we propose learning an optimized controller using a DNN that fuses multiple controllers. The network learns a robust controller with online trajectory filtering, which suppresses noisy trajectories and imperfections of individual controllers. The result is a network that is able to learn a good fusion of filtered trajectories from different controllers leading to significant improvements in overall performance. We compare our trained network to controllers it has learned from, end-to-end baselines and human pilots in a realistic simulation; our network beats all baselines in extensive experiments and approaches the performance of a professional human pilot.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: A Supplementary Materialsmentioning

confidence: 99%

See 1 more Smart Citation

Learning a Controller Fusion Network by Online Trajectory Filtering for Vision-Based UAV Racing

Müller

Casser

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The rest of the terms in Eq.1 are smoothness terms: color term 3 � , �, depth term 4 � , �, and color gradient term 5 � , �. Similar to (43), the color term 3 � , � is defined by a truncated seam-hiding pairwise cost first introduced in (44): 3…”

Section: Augmented Background Synthesismentioning

confidence: 99%

AADS: Augmented autonomous driving simulation using data-driven algorithms

et al. 2019

View full text Add to dashboard Cite

Simulation systems have become an essential component in the development and validation of autonomous driving technologies. The prevailing state-of-the-art approach for simulation is to use game engines or high-fidelity computer graphics (CG) models to create driving scenarios. However, creating CG models and vehicle movements (a.k.a. the assets for simulation) remains a manual task that can be costly and time-consuming. In addition, the fidelity of CG images still lacks the richness and authenticity of real-world images and using these CG images for training leads to degraded performance.In this paper we present a novel approach to address these issues: Augmented Autonomous Driving Simulation (AADS). Our formulation augments real-world pictures with a simulated traffic flow to create photo-realistic simulation images and renderings. More specifically, we use LiDAR and cameras to scan street scenes. From the acquired trajectory data, we generate highly plausible traffic flows for cars and pedestrians and compose them into the background. The composite images can be re-synthesized with different viewpoints and sensor models (camera or LiDAR). The resulting images are photo-realistic, fully annotated, and ready for end-to-end training and testing of autonomous driving systems from perception to planning. We explain our system design and validate our algorithms with a number of autonomous driving tasks from detection to segmentation and predictions.Compared to traditional approaches, our method offers unmatched scalability and realism. Scalability is particularly important for AD simulation and we believe the complexity and diversity of the real world cannot be realistically captured in a virtual environment. Our augmented approach combines the flexibility of a virtual environment (e.g., vehicle movements) with the richness of the real world to allow effective simulation of any location on earth. Summary

show abstract

“…Multi-person datasets were created by employing video games for pedestrian detection [29] and pose tracking [12]. Similarly, [30] develop a simulation environment in a game engine, including virtual humans. Related to our approach, [38] augment real training images similar to the ones in [44] but with multiple synthetic humans occluding each other.…”

Section: Related Workmentioning

confidence: 99%

Learning to Train with Synthetic Humans

Hoffmann

Tzionas

Black

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Neural networks need big annotated datasets for training. However, manual annotation can be too expensive or even unfeasible for certain tasks, like multi-person 2D pose estimation with severe occlusions. A remedy for this is synthetic data with perfect ground truth. Here we explore two variations of synthetic data for this challenging problem; a dataset with purely synthetic humans and a real dataset augmented with synthetic humans. We then study which approach better generalizes to real data, as well as the influence of virtual humans in the training loss. Using the augmented dataset, without considering synthetic humans in the loss, leads to the best results. We observe that not all synthetic samples are equally informative for training, while the informative samples are different for each training stage. To exploit this observation, we employ an adversarial student-teacher framework; the teacher improves the student by providing the hardest samples for its current state as a challenge. Experiments show that the student-teacher framework outperforms normal training on the purely synthetic dataset.

show abstract

Sim4CV: A Photo-Realistic Simulator for Computer Vision Applications

Cited by 142 publications

References 54 publications

Learning a Controller Fusion Network by Online Trajectory Filtering for Vision-Based UAV Racing

Learning a Controller Fusion Network by Online Trajectory Filtering for Vision-Based UAV Racing

AADS: Augmented autonomous driving simulation using data-driven algorithms

Learning to Train with Synthetic Humans

Contact Info

Product

Resources

About