Neural 3D holography

Choi, Suyeon; Gopakumar, Manu; Peng, Yifan; Kim, Jonghyun; Wetzstein, Gordon

doi:10.1145/3478513.3480542

Cited by 100 publications

(31 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Second is the smooth phase distribution assigned to the image contents in the diffraction calculation for ground truth hologram synthesis, which helps boost easy learning of diffraction features with small numerical aperture propagation. But the lack of randomness [25,26] in such smooth phase strategy degrade the comfort of holographic 3D viewing experiences (see Supplement 1 for more details).…”

Section: From Picture To 3d Holography: End-to-end Learning Of Real-t...mentioning

confidence: 99%

From picture to 3D hologram: end-to-end learning of real-time 3D photorealistic hologram generation from 2D image input

Chang¹,

Dai²,

Zhu³

et al. 2023

Opt. Lett.

View full text Add to dashboard Cite

In this Letter, we demonstrate a deep-learning-based method capable of synthesizing a photorealistic 3D hologram in real-time directly from the input of a single 2D image. We design a fully automatic pipeline to create large-scale datasets by converting any collection of real-life images into pairs of 2D images and corresponding 3D holograms and train our convolutional neural network (CNN) end-to-end in a supervised way. Our method is extremely computation-efficient and memory-efficient for 3D hologram generation merely from the knowledge of on-hand 2D image content. We experimentally demonstrate speckle-free and photorealistic holographic 3D displays from a variety of scene images, opening up a way of creating real-time 3D holography from everyday pictures. © 2023 Optical Society of America

show abstract

Section: From Picture To 3d Holography: End-to-end Learning Of Real-t...mentioning

confidence: 99%

From picture to 3D hologram: end-to-end learning of real-time 3D photorealistic hologram generation from 2D image input

Chang¹,

Dai²,

Zhu³

et al. 2023

Opt. Lett.

View full text Add to dashboard Cite

show abstract

“…Light field (LF) cameras record both intensity and directions of light rays, and enable various applications such as depth perception [24,28,31], view rendering [3,51,65], virtual reality [10,73], and 3D reconstruction [6,76]. However, due to the inherent spatial-angular trade-off [81], an LF camera can either provide dense angular samplings with low-resolution (LR) sub-aperture images (SAIs), or capture high-resolution (HR) SAIs with sparse angular sampling.…”

Section: Introductionmentioning

confidence: 99%

Learning Non-Local Spatial-Angular Correlation for Light Field Image Super-Resolution

Liang¹,

Wang²,

Wang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Exploiting spatial-angular correlation is crucial to light field (LF) image super-resolution (SR), but is highly challenging due to its non-local property caused by the disparities among LF images. Although many deep neural networks (DNNs) have been developed for LF image SR and achieved continuously improved performance, existing methods cannot well leverage the long-range spatialangular correlation and thus suffer a significant performance drop when handling scenes with large disparity variations. In this paper, we propose a simple yet effective method to learn the non-local spatial-angular correlation for LF image SR. In our method, we adopt the epipolar plane image (EPI) representation to project the 4D spatialangular correlation onto multiple 2D EPI planes, and then develop a Transformer network with repetitive self-attention operations to learn the spatial-angular correlation by modeling the dependencies between each pair of EPI pixels. Our method can fully incorporate the information from all angular views while achieving a global receptive field along the epipolar line. We conduct extensive experiments with insightful visualizations to validate the effectiveness of our method. Comparative results on five public datasets show that our method not only achieves state-of-the-art SR performance, but also performs robust to disparity variations. Code is publicly available at https://github.com/ ZhengyuLiang24/EPIT.

show abstract

“…To overcome these limitations, substantial efforts have been devoted to addressing the artifacts arising from binary modulation. End-to-end [62][63][64][65] and deep-learning-based methods [43,66] take advantage of scientific understanding (mathematical models) or observations (training datasets) of a specific system to eliminate the underlying artifacts. Rather than improving the fidelity of individual frames, temporal multiplexing methods [48,49,58] exploit the statistical properties of speckle patterns to improve the projection quality via temporal averaging.…”

Section: Introductionmentioning

confidence: 99%

High-Fidelity and High-Speed Wavefront Shaping through Complex Media via Sparsity-Constrained Optimization

Yu¹,

You²

2023

Preprint

View full text Add to dashboard Cite

Achieving high-precision light manipulation is crucial for delivering information through complex media with high fidelity. However, existing spatial light modulation devices face a fundamental tradeoff between speed and accuracy, limiting their use in various real-time and qualitydemanding applications. To address this challenge, we propose a physics-based sparsity-constrained optimization framework for enhancing projection quality through complex media at a full DMD frame rate of 22 kHz. By addressing the limited degrees of freedom, scattering effect, and ill-posed and ill-conditioned nature of the inverse problem, our method achieves solutions with higher feasibility, optimality, and better numerical stability simultaneously. In addition, our method is system-agnositc and generalizable, showing consistent performance across different types of complex media. These results demonstrate the potential of our method in paving the way for high-fidelity and highspeed wavefront shaping in complex media, enabling a wide range of applications, such as non-invasive deep brain imaging, high-speed holographic optogenetics, and miniaturized fiber-based 3D printing devices.

show abstract

Neural 3D holography

Cited by 100 publications

References 39 publications

From picture to 3D hologram: end-to-end learning of real-time 3D photorealistic hologram generation from 2D image input

From picture to 3D hologram: end-to-end learning of real-time 3D photorealistic hologram generation from 2D image input

Learning Non-Local Spatial-Angular Correlation for Light Field Image Super-Resolution

High-Fidelity and High-Speed Wavefront Shaping through Complex Media via Sparsity-Constrained Optimization

Contact Info

Product

Resources

About