Inverse Transport Networks

Che, Chengqian; Luan, Fujun; Zhao, Shuang; Bala, Kavita; Gkioulekas, Ioannis

doi:10.48550/arxiv.1809.10820

Cited by 10 publications

(7 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A loss can then be applied on the predicted parameters, or they can be used by the physical model to create additional output(s). For example, in the work of Che et al [67], the physical scene parameters necessary to reconstruct an image with a renderer, such as shape, material, and illumination, are predicted and compared to known ground truth parameter values. In general, access to ground truth Fig.…”

Section: Physics Regularizationmentioning

confidence: 99%

Physics vs. Learned Priors: Rethinking Camera and Algorithm Design for Task-Specific Imaging

Klinghoffer¹,

Somasundaram²,

Tiwary³

et al. 2022

Preprint

View full text Add to dashboard Cite

Cameras were originally designed using physics-based heuristics to capture aesthetic images. In recent years, there has been a transformation in camera design from being purely physics-driven to increasingly data-driven and task-specific. In this paper, we present a framework to understand the building blocks of this nascent field of end-to-end design of camera hardware and algorithms. As part of this framework, we show how methods that exploit both physics and data have become prevalent in imaging and computer vision, underscoring a key trend that will continue to dominate the future of task-specific camera design. Finally, we share current barriers to progress in end-to-end design, and hypothesize how these barriers can be overcome.

show abstract

Section: Physics Regularizationmentioning

confidence: 99%

Physics vs. Learned Priors: Rethinking Camera and Algorithm Design for Task-Specific Imaging

Klinghoffer¹,

Somasundaram²,

Tiwary³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…A prime example is Optical Diffusion Tomography (ODT) [20,21,22], which uses non-ionizing near-infrared light. It is worth noting the work by Che et al [23] which departs from physics-based approaches into the realm of machine-learning.…”

Section: Why Passive Tomography?mentioning

confidence: 99%

Multi-view polarimetric scattering cloud tomography and retrieval of droplet size

Levis¹,

Schechner²,

Davis³

et al. 2020

Preprint

View full text Add to dashboard Cite

Tomography aims to recover a three-dimensional (3D) density map of a medium or an object. In medical imaging, it is extensively used for diagnostics via X-ray computed tomography (CT). Optical diffusion tomography is an alternative to X-ray CT that uses multiply scattered light to deliver coarse density maps for soft tissues. We define and derive tomography of cloud droplet distributions via passive remote sensing. We use multi-view polarimetric images to fit a 3D polarized radiative transfer (RT) forward model. Our motivation is 3D volumetric probing of vertically-developed convectively-driven clouds that are ill-served by current methods in operational passive remote sensing. These techniques are based on strictly 1D RT modeling and applied to a single cloudy pixel, where cloud geometry is assumed to be that of a plane-parallel slab. Incident unpolarized sunlight, once scattered by cloud-droplets, changes its polarization state according to droplet size. Therefore, polarimetric measurements in the rainbow and glory angular regions can be used to infer the droplet size distribution. This work defines and derives a framework for a full 3D tomography of cloud droplets for both their mass concentration in space and their distribution across a range of sizes. This 3D retrieval of key microphysical properties is made tractable by our novel approach that involves a restructuring and differentiation of an open-source polarized 3D RT code to accommodate a special two-step optimization technique. Physically-realistic synthetic clouds are used to demonstrate the methodology with rigorous uncertainty quantification.

show abstract

“…A number of recent deep inverse rendering methods have incorporated in-network, differentiable rendering layers that are customized for simple settings: faces [51,56], planar surfaces [18,37], single objects [40,38]. Some recent work has proposed differentiable general-purpose global illumination renderers [35,15]; unlike our more specialized, fast rendering layer, these are too expensive to use for neural network training.…”

Section: Differentiable Renderingmentioning

confidence: 99%

Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image

Shafiei

Ramamoorthi

et al. 2019

Preprint

View full text Add to dashboard Cite

We propose a deep inverse rendering framework for indoor scenes. From a single RGB image of an arbitrary indoor scene, we create a complete scene reconstruction, estimating shape, spatially-varying lighting, and spatiallyvarying, non-Lambertian surface reflectance. To train this network, we augment the SUNCG indoor scene dataset with real-world materials and render them with a fast, highquality, physically-based GPU renderer to create a largescale, photorealistic indoor dataset. Our inverse rendering network incorporates physical insights -including a spatially-varying spherical Gaussian lighting representation, a differentiable rendering layer to model scene appearance, a cascade structure to iteratively refine the predictions and a bilateral solver for refinement -allowing us to jointly reason about shape, lighting, and reflectance. Experiments show that our framework outperforms previous methods for estimating individual scene components, which also enables various novel applications for augmented reality, such as photorealistic object insertion and material editing. Code and data will be made publicly available.

show abstract

Inverse Transport Networks

Cited by 10 publications

References 60 publications

Physics vs. Learned Priors: Rethinking Camera and Algorithm Design for Task-Specific Imaging

Physics vs. Learned Priors: Rethinking Camera and Algorithm Design for Task-Specific Imaging

Multi-view polarimetric scattering cloud tomography and retrieval of droplet size

Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image

Contact Info

Product

Resources

About