Motion deblurring using spatiotemporal phase aperture coding

Elmalem, Shay; Giryes, Raja; Marom, Emanuel

doi:10.1364/optica.399533

Cited by 14 publications

(15 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Under this model, the PSFs are calculated on a grid and then convolved with a weighted image, and then summed to generate the modulated image I mod (n, m; C l ), i.e. : (10) where * is a 2D convolution, C l is the color channel, ω k (n, m) is the window weighting function associated with PSF k which is the spatially variant PSF calculated at a chosen grid point and I 0 (n, m; C l ) is the input image at pixel (n, m) and wavelength C l . The amount of chosen kernels (K) creates a trade-off between computational efficiency and accuracy.…”

Section: End-to-end Designmentioning

confidence: 99%

“…The commonly used geometrical optics (ray tracing) method is a crude approximation of wave optics, yet it provides powerful tools to design thick elements and complex stacks of lenses. However, geometrical optics is not suitable for applications such as phase retrieval [1][2][3][4], adaptive optics [5,6], PSF engineering [7][8][9][10], or more generally, any application that requires either correctly modelling the diffraction-limited PSF or employing optical elements that manipulate the phase, amplitude, or polarization of light. Such applications require Fourier optics [11], which can account for diffraction effects, but is usually limited to thin-element and paraxial approximations.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Diffractive optical system design by cascaded propagation

Ferdman,

Saguy,

Alalouf

et al. 2022

Preprint

View full text Add to dashboard Cite

Modern design of complex optical systems relies heavily on computational tools. These typically utilize geometrical optics as well as Fourier optics, which enables the use of diffractive elements to manipulate light with features on the scale of a wavelength. Fourier optics is typically used for designing thin elements, placed in the system's aperture, generating a shiftinvariant Point Spread Function (PSF). A major bottleneck in applying Fourier Optics in many cases of interest, e.g. when dealing with multiple, or out-of-aperture elements, comes from numerical complexity. In this work, we propose and implement an efficient and differentiable propagation model based on the Collins integral, which enables the optimization of diffraction optical systems with unprecedented design freedom using backpropagation. We demonstrate the applicability of our method, numerically and experimentally, by engineering shift-variant PSFs via thin plate elements placed in arbitrary planes inside complex imaging systems, performing cascaded optimization of multiple planes, and designing optimal machine-vision systems by deep learning.

show abstract

Section: End-to-end Designmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Diffractive optical system design by cascaded propagation

Ferdman,

Saguy,

Alalouf

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Both of these approaches are limited to the reconstruction of a single image. Dynamic phase coding in the lens aperture for motion coding was presented by Elmalem et al [13] for motion deblurring. This coding embeds motion cues in the intermediate image for improved deblurring performance.…”

Section: Related Workmentioning

confidence: 99%

Video Reconstruction from a Single Motion Blurred Image using Learned Dynamic Phase Coding

Yosef¹,

Elmalem²,

Giryes³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Video reconstruction from a single motion-blurred image is a challenging problem, which can enhance existing cameras' capabilities. Recently, several works addressed this task using conventional imaging and deep learning. Yet, such purely-digital methods are inherently limited, due to direction ambiguity and noise sensitivity. Some works proposed to address these limitations using non-conventional image sensors, however, such sensors are extremely rare and expensive. To circumvent these limitations with simpler means, we propose a hybrid optical-digital method for video reconstruction that requires only simple modifications to existing optical systems. We use a learned dynamic phase-coding in the lens aperture during the image acquisition to encode the motion trajectories, which serve as prior information for the video reconstruction process. The proposed computational camera generates a sharp frame burst of the scene at various frame rates from a single coded motion-blurred image, using an image-to-video convolutional neural network. We present advantages and improved performance compared to existing methods, using both simulations and a real-world camera prototype.

show abstract

“…In 2020 Elmalem made use of a joint optical-digital spatio-temporal phase aperture coding scheme to implement motion deblurring. In this work dynamic phase coding was implemented in the aperture during image acquisition [19]. The advantage of Elmalem's approach is that it could perform encoding without any limitation on the direction of motion and without sacrificing light efficiency.…”

Section: Introductionmentioning

confidence: 99%

Digital coded exposure formation of frames from event-based imagery

Gothard¹,

Jones²,

Green³

et al. 2022

Neuromorph. Comput. Eng.

View full text Add to dashboard Cite

Event-driven neuromorphic imagers have a number of attractive properties including low-power consumption, high dynamic range, the ability to detect fast events, low memory consumption and low band-width requirements. One of the biggest challenges with using event-driven imagery is that the field of event data processing is still embryonic. In contrast, decades worth of effort have been invested in the analysis of frame-based imagery. Hybrid approaches for applying established frame-based analysis techniques to event-driven imagery have been studied since event-driven imagers came into existence. However, the process for forming frames from event-driven imagery has not been studied in detail. This work presents a principled digital coded exposure approach for forming frames from event-driven imagery that is inspired by the physics exploited in a conventional camera featuring a shutter. The technique described in this work provides a fundamental tool for understanding the temporal information content that contributes to the formation of a frame from event-driven imagery data. Event-driven imagery allows for the application of arbitrary virtual digital shutter functions to form the final frame on a pixel-by-pixel basis. The proposed approach allows for the careful control of the spatio-temporal information that is captured in the frame. Furthermore, unlike a conventional physical camera, event-driven imagery can be formed into any variety of possible frames in post-processing after the data is captured. Furthermore, unlike a conventional physical camera, coded-exposure virtual shutter functions can assume arbitrary values including positive, negative, real, and complex values. The coded exposure approach also enables the ability to perform applications of industrial interest such as digital stroboscopy without any additional hardware. The ability to form frames from event-driven imagery in a principled manner opens up new possibilities in the ability to use conventional frame-based image processing techniques on event-driven imagery.

show abstract

Motion deblurring using spatiotemporal phase aperture coding

Cited by 14 publications

References 36 publications

Diffractive optical system design by cascaded propagation

Diffractive optical system design by cascaded propagation

Video Reconstruction from a Single Motion Blurred Image using Learned Dynamic Phase Coding

Digital coded exposure formation of frames from event-based imagery

Contact Info

Product

Resources

About