A vision system using depth inpainting for virtual content reconstruction in Augmented Reality

Voronin, Viacheslav; Gapon, Nikolay; Khamidullin, Iliy; Tokareva, Olga; Zelensky, Alexander A.; Semenishchev, Evgeny A.

doi:10.1117/12.2642007

Cited by 3 publications

(7 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, threshold detection occurs, leaving only those areas that need to be restored. At the first step, for each pixel of the border , using the inversion method, the shape of the region for similarity search is adaptively determined, which is formed by combining two adjacent homogeneous sub-regions in the direction of the gradient maximum [5].…”

Section: Reconstruction Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Defect detection and removal for depth map quality enhancement in manufacturing with deep learning

Gapon

Voronin

Sizyakin

et al. 2023

Dimensional Optical Metrology and Inspection for Practical Applications XII

View full text Add to dashboard Cite

A system for determination the distance from the robot to the scene is useful for object tracking, and 3-D reconstruction may be desired for many manufacturing and robotic tasks. While the robot is processing materials, such as welding parts, milling, drilling, fragments of materials fall on the camera installed on the robot, introducing unnecessary information when building a depth map, as well as the emergence of new lost areas, which leads to incorrect determination of the size of objects. There is a problem comprising a decrease in the accuracy of planning the movement trajectory caused by incorrect sections on the depth map because of incorrect distance determination of objects. We present a two-stage approach combining defect detection and depth reconstruction algorithms. The first step is image defects detection based on convolutional auto-encoder (U-Net) and deep feature fusion network (DFFN-Net). The second step is a depth map reconstruction with the exemplar-based and the anisotropic gradient concepts. The proposed modified block fusion algorithm uses a local image descriptor obtained by an automatic encoder for image reconstruction, which extracts image features and depth maps using a decoding network. Our technique outperforms the state-of-the-art methods quantitatively in reconstruction accuracy on RGB-D benchmark for evaluating manufacturing vision systems.

show abstract

Section: Reconstruction Methodsmentioning

confidence: 99%

“…On the second step calculates a priority [5] value of for each border pixel value, which consists of two multipliers:…”

Section: Reconstruction Methodsmentioning

confidence: 99%

Defect detection and removal for depth map quality enhancement in manufacturing with deep learning

Gapon

Voronin

Sizyakin

et al. 2023

Dimensional Optical Metrology and Inspection for Practical Applications XII

View full text Add to dashboard Cite

show abstract

“…At the first step, for each pixel of the border , using the inversion method, the shape of the region for similarity search is adaptively determined [10].…”

Section: Inpaintingmentioning

confidence: 99%

“…-current pixel on the border of available pixels; -confidence factor [10]; -gradient factor [10]. Priority calculation allows us to give more weight to pixels on the brightness differences (borders), thus restoring them in the first place [12].…”

Section: Inpaintingmentioning

confidence: 99%

Deep learning-based depth map defect removal for industrial applications

Voronin,

Gapon,

Zhdanova

et al. 2023

Optical Metrology and Inspection for Industrial Applications X

View full text Add to dashboard Cite

A system for determining the distance from the robot to the scene is useful for object tracking, and 3-D reconstructions may be desired for many manufacturing and robotic tasks. While the robot is processing materials, such as welding parts, milling, drilling, etc., fragments of materials fall on the camera installed on the robot, introducing unnecessary information when building a depth map, as well as the emergence of new lost areas, which leads to incorrect determination of the size of objects. There is a problem comprising a decrease in the accuracy of planning the movement trajectory caused by wrong sections on the depth map because of erroneous distance determination to objects. We present an approach combining defect detection and depth reconstruction algorithms. The first step for image defect detection is based on a convolutional auto-encoder (U-Net). The second step is a depth map reconstruction using a spatial reconstruction based on a geometric model with contour and texture analysis. We apply contour restoration and texture synthesis for image reconstruction. A method is proposed for restoring the boundaries of objects in an image based on constructing a composite curve by cubic splines. Our technique outperforms the state-of-the-art methods quantitatively in reconstruction accuracy on the RGB-D benchmark for evaluating manufacturing vision systems.

show abstract

“…It can also remove various types of distortion, including text, blocks, noise, scratches, lines, etc. [1][2][3][4][5][6][7]. Image restoration is the process of restoring the missing area of an image.…”

Section: Introductionmentioning

confidence: 99%

Image inpainting with deep adaptive generative models for multimedia computing

Voronin,

Gapon,

Zhdanova

et al. 2023

Optoelectronic Imaging and Multimedia Technology X

View full text Add to dashboard Cite

Image inpainting is a challenging task where large missing regions must be filled based on the available data. It is an interesting new research topic in multimedia computing and image processing. This paper proposes a novel deep learning reconstruction method using an adaptive generative adversarial neural network (aGAN). The key feature of the proposed network is the dynamic estimation of the trend of the loss function. This estimate finds the optimal balance between adversarial and pixel-by-pixel losses. We introduced the coefficient of the approximating curve, which allows us to estimate the trend dynamics of the loss function. Depending on the value of this curve, we adjust the weight of the pixelby-pixel loss to make the restored area sharper. The proposed method allows a higher sharpness of the reconstructed area and a more stable learning process. Experiments on the dataset show that our method successfully predicts multimedia information in missing regions and significantly outperforms the state-of-the-art methods.

show abstract

A vision system using depth inpainting for virtual content reconstruction in Augmented Reality

Cited by 3 publications

References 21 publications

Defect detection and removal for depth map quality enhancement in manufacturing with deep learning

Defect detection and removal for depth map quality enhancement in manufacturing with deep learning

Deep learning-based depth map defect removal for industrial applications

Image inpainting with deep adaptive generative models for multimedia computing

Contact Info

Product

Resources

About