AutoSimulate: (Quickly) Learning Synthetic Data Generation

Behl, Harkirat Singh; Baydin, Atılım Güneş; Gal, Ran; Torr, Philip H. S.; Vineet, Vibhav

doi:10.1007/978-3-030-58542-6_16

Cited by 16 publications

(20 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our approach does not use scene grammar. Most similar to our work is the work of Auto-Simulate [4] that proposed a local approximation of the bilevel optimization to efficiently solve the problem. However, since they optimized non-differentiable simulators like Blender [10] and Arnold [21], they used REINFORCE-based [47] gradient update.…”

Section: Related Workmentioning

confidence: 94%

“…Learning simulator parameters. Related works in this space focus on learning non-differentiable simulator parameters for e.g., learning-to-simulate (LTS) [42], Meta-Sim [30], Meta-Sim2 [11], Auto-Sim [4], and others [51,17,32]. Our work in contrast has two differences: (i) a difference in the renderer used (NeRF vs traditional rendering engines), and (ii) a difference in the optimization approach.…”

Section: Related Workmentioning

confidence: 99%

“…Optimizing synthetic data generation Our goal is to optimize over the rendering distribution p ψ such that training an object detection model on D train leads to good performance on D val . We formulate this problem as a bi-level optimization [8,15,4] as below:…”

Section: Neural-simmentioning

confidence: 99%

“…al. [4] developed a technique for numerical gradient computation based on local approximation of the bi-level optimization. Without going into their derivation, we borrow the gradient term for the outer update, which at time step t takes the form:…”

Section: Neural-simmentioning

confidence: 99%

“…∇ T V is computed using the conjugate gradient method [4]. However, [4] treated the data generation as a black box and used REINFORCE [48] to compute the approximate gradient because they used non-differentiable renderers for data generation. However, REINFORCE is considered noisy process and is known to lead to high-variance estimates of gradients.…”

Section: Neural-simmentioning

confidence: 99%

See 4 more Smart Citations

Neural-Sim: Learning to Generate Training Data with NeRF

Ge¹,

Behl²,

Xu³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Training computer vision models usually requires collecting and labeling vast amounts of imagery under a diverse set of scene configurations and properties. This process is incredibly time-consuming, and it is challenging to ensure that the captured data distribution maps well to the target domain of an application scenario. Recently, synthetic data has emerged as a way to address both of these issues. However, existing approaches either require human experts to manually tune each scene property or use automatic methods that provide little to no control; this requires rendering large amounts of random data variations, which is slow and is often suboptimal for the target domain. We present the first fully differentiable synthetic data pipeline that uses Neural Radiance Fields (NeRFs) in a closed-loop with a target application's loss function. Our approach generates data on-demand, with no human labor, to maximize accuracy for a target task. We illustrate the effectiveness of our method on synthetic and real-world object detection tasks. We also introduce a new "YCB-in-the-Wild" dataset and benchmark that provides a test scenario for object detection with varied poses in real-world environments. Code and data could be found at https://github.com/gyhandy/Neural-Sim-NeRF.

show abstract

Section: Related Workmentioning

confidence: 94%