Superpixel Soup: Monocular Dense 3D Reconstruction of a Complex Dynamic Scene

Kumar, Suryansh; Dai, Yuchao; Li, Hongdong

doi:10.1109/tpami.2019.2955131

Cited by 35 publications

(24 citation statements)

References 64 publications

(143 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly, rewriting the Eq:(15) treating S as variable. 20) In contrast to the previous form, the solution to Eq:(20) is not straight forward. To obtain a closed form solution to this problem, lets define a soft-thresholding function S τ (σ) = sign(σ).max(|σ| − τ, 0).…”

Section: Solutionmentioning

confidence: 97%

See 1 more Smart Citation

Non-Rigid Structure from Motion: Prior-Free Factorization Method Revisited

Kumar

2020

2020 IEEE Winter Conference on Applications of Computer Vision (WACV)

Self Cite

View full text Add to dashboard Cite

A simple prior free factorization algorithm [9] is quite often cited work in the field of Non-Rigid Structure from Motion (NRSfM). The benefit of this work lies in its simplicity of implementation, strong theoretical justification to the motion and structure estimation, and its invincible originality. Despite this, the prevailing view is, that it performs exceedingly inferior to other methods on several benchmark datasets [14, 1]. However, our subtle investigation provides some empirical statistics which made us think against such views. The statistical results we obtained supersedes Dai et al. [9] originally reported results on the benchmark datasets by a significant margin under some elementary changes in their core algorithmic idea [9]. Now, these results not only exposes some unrevealed areas for research in NRSfM but also give rise to new mathematical challenges for NRSfM researchers. We argue that by properly utilizing the well-established assumptions about a non-rigidly deforming shape i.e, it deforms smoothly over frames [27] and it spans a low-rank space, the simple prior-free idea can provide results which is comparable to the best available algorithms. In this paper, we explore some of the hidden intricacies missed by Dai et. al. work [9] and how some elementary measures and modifications can enhance its performance, as high as approx. 18% on the benchmark dataset. The improved performance is justified and empirically verified by extensive experiments on several datasets. We believe our work has both practical and theoretical importance for the development of better NRSfM algorithms.

show abstract

Section: Solutionmentioning

confidence: 97%

“…Non-rigid Structure from Motion (NRSfM) is a wellknown problem in geometric computer vision [5,1,9,20,18]. The goal of this problem is to reconstruct 3D structure of a deforming object using multiple frames.…”

Section: Introductionmentioning

confidence: 99%

Non-Rigid Structure from Motion: Prior-Free Factorization Method Revisited

Kumar

2020

2020 IEEE Winter Conference on Applications of Computer Vision (WACV)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Carranza et al [Carranza et al 2003] uses a multi-view system to recover 3D models from silhouettes for synthesizing novel views from arbitrary perspectives. Kumar et al [Kumar et al 2021] uses a monocular video as input to recover a dynamic scene by a conventional optimization approach with two assumptions of scene and deformation, which is 1) The dynamic scene can be approximated by multiple piece-wise planar surfaces with rigid motions, 2) the deformation of the scene is locally rigid but global as-rigid-as possible. However, these methods above have limited ability to model and manipulate the complicated scene geometry for further visual effect rendering.…”

Section: Related Workmentioning

confidence: 99%

Editable free-viewpoint video using a layered neural representation

Zhang

Liu

et al. 2021

ACM Trans. Graph.

View full text Add to dashboard Cite

Generating free-viewpoint videos is critical for immersive VR/AR experience, but recent neural advances still lack the editing ability to manipulate the visual perception for large dynamic scenes. To fill this gap, in this paper, we propose the first approach for editable free-viewpoint video generation for large-scale view-dependent dynamic scenes using only 16 cameras. The core of our approach is a new layered neural representation, where each dynamic entity, including the environment itself, is formulated into a spatiotemporal coherent neural layered radiance representation called ST-NeRF. Such a layered representation supports manipulations of the dynamic scene while still supporting a wide free viewing experience. In our ST-NeRF, we represent the dynamic entity/layer as a continuous function, which achieves the disentanglement of location, deformation as well as the appearance of the dynamic entity in a continuous and self-supervised manner. We propose a scene parsing 4D label map tracking to disentangle the spatial information explicitly and a continuous deform module to disentangle the temporal motion implicitly. An object-aware volume rendering scheme is further

show abstract

“… Kumar, Dai & Li (2019) illustrates a technique to build 3D reconstruction of a complex dynamic scene using two frames by applying super-pixel over-segmentation to the image. A generically dynamic (hence non-rigid) scene with a piecewise planar and rigid approximation are presented.…”

Section: State Of the Artmentioning

confidence: 99%

A novel no-sensors 3D model reconstruction from monocular video frames for a dynamic environment

Fathy

Hassan

Sheta

et al. 2021

PeerJ Computer Science

View full text Add to dashboard Cite

Occlusion awareness is one of the most challenging problems in several fields such as multimedia, remote sensing, computer vision, and computer graphics. Realistic interaction applications are suffering from dealing with occlusion and collision problems in a dynamic environment. Creating dense 3D reconstruction methods is the best solution to solve this issue. However, these methods have poor performance in practical applications due to the absence of accurate depth, camera pose, and object motion.This paper proposes a new framework that builds a full 3D model reconstruction that overcomes the occlusion problem in a complex dynamic scene without using sensors’ data. Popular devices such as a monocular camera are used to generate a suitable model for video streaming applications. The main objective is to create a smooth and accurate 3D point-cloud for a dynamic environment using cumulative information of a sequence of RGB video frames. The framework is composed of two main phases. The first uses an unsupervised learning technique to predict scene depth, camera pose, and objects’ motion from RGB monocular videos. The second generates a frame-wise point cloud fusion to reconstruct a 3D model based on a video frame sequence. Several evaluation metrics are measured: Localization error, RMSE, and fitness between ground truth (KITTI’s sparse LiDAR points) and predicted point-cloud. Moreover, we compared the framework with different widely used state-of-the-art evaluation methods such as MRE and Chamfer Distance. Experimental results showed that the proposed framework surpassed the other methods and proved to be a powerful candidate in 3D model reconstruction.

show abstract

Superpixel Soup: Monocular Dense 3D Reconstruction of a Complex Dynamic Scene

Cited by 35 publications

References 64 publications

Non-Rigid Structure from Motion: Prior-Free Factorization Method Revisited

Non-Rigid Structure from Motion: Prior-Free Factorization Method Revisited

Editable free-viewpoint video using a layered neural representation

A novel no-sensors 3D model reconstruction from monocular video frames for a dynamic environment

Contact Info

Product

Resources

About