4D Match Trees for Non-rigid Surface Alignment

IEEE Trans. on Image Process.

Kim

Hilton

2019

Self Cite

A common problem in wide-baseline matching is the sparse and non-uniform distribution of correspondences when using conventional detectors such as SIFT, SURF, FAST, A-KAZE and MSER. In this paper we introduce a novel segmentation based feature detector (SFD) that produces an increased number of accurate features for wide-baseline matching. A multi-scale SFD is proposed using bilateral image decomposition to produce a large number of scale-invariant features for wide-baseline reconstruction. All input images are over-segmented into regions using any existing segmentation technique like Watershed, Mean-shift, and SLIC. Feature points are then detected at the intersection of the boundaries of three or more regions. The detected feature points are local maxima of the image function. The key advantage of feature detection based on segmentation is that it does not require global threshold setting and can therefore detect features throughout the image. A comprehensive evaluation demonstrates that SFD gives an increased number of features which are accurately localised and matched between wide-baseline camera views; the number of features for a given matching error increases by a factor of 3-5 compared to SIFT; feature detection and matching performance is maintained with increasing baseline between views; multi-scale SFD improves matching performance at varying scales. Application of SFD to sparse multi-view wide-baseline reconstruction demonstrates a factor of ten increase in the number of reconstructed points with improved scene coverage compared to SIFT/MSER/A-KAZE. Evaluation against ground-truth shows that SFD produces an increased number of wide-baseline matches with reduced error.

Section: Segmentationmentioning

confidence: 78%

MSFD: Multi-Scale Segmentation-Based Feature Detection for Wide-Baseline Scene Reconstruction

IEEE Trans. on Image Process.

Kim

Hilton

2019

Self Cite

“…Qualitative evaluation: For comparative evaluation we use:(a) state-of-the-art dense flow algorithm Deepflow [35]; (b) dense flow without light-field consistency (DFwLF) in eq. 2; (c) a recent algorithm for alignment of partial surfaces (4DMatch) [11] and (d) Simple flow [34]. Qualitative results against DFwLF, 4DMatch, Deepflow and Simpleflow shown in Figure 12 indicate that the propagated colour map does not remain consistent across the sequence for large Figure 11.…”

Section: Results and Performance Evaluationmentioning

confidence: 99%

“…However, these approaches assume a reconstruction of the full non-rigid object surface at each time frame and do not easily extend to 4D alignment of partial surface reconstructions or depth maps. Recent work obtained reliable temporal alignment of partial surfaces for complex dynamic scenes [11,12,13,14]. Dynamic Fusion [15] was introduced for 4D modelling from depth image sequences integrating temporal observations of non-rigid shape to resolve fine detail.…”

Section: Introductionmentioning

confidence: 99%

4D Temporally Coherent Light-Field Video

2017 International Conference on 3D Vision (3DV)

Volino

Guillemaut

et al. 2017

Self Cite

Light-field video has recently been used in virtual and augmented reality applications to increase realism and immersion. However, existing light-field methods are generally limited to static scenes due to the requirement to acquire a dense scene representation. The large amount of data and the absence of methods to infer temporal coherence pose major challenges in storage, compression and editing compared to conventional video. In this paper, we propose the first method to extract a spatio-temporally coherent light-field video representation. A novel method to obtain Epipolar Plane Images (EPIs) from a spare lightfield camera array is proposed. EPIs are used to constrain scene flow estimation to obtain 4D temporally coherent representations of dynamic light-fields. Temporal coherence is achieved on a variety of light-field datasets. Evaluation of the proposed light-field scene flow against existing multiview dense correspondence approaches demonstrates a significant improvement in accuracy of temporal coherence.

“…Flow from the joint estimation is evaluated against stateof-the-art methods: (a) Dense flow algorithms DCflow [57] and Deepflow [54]; (b) Scene flow methods PRSM [52]; and (c) Non-sequential alignment of partial surfaces 4DMatch [38] (requires a prior 3D mesh of the object as input for 4D reconstruction). The key-frames of sequence are coloured and the colour is propagated using dense flow from the joint optimisation throughout the sequence.…”

Section: Motion Evaluationmentioning

confidence: 99%

U4D: Unsupervised 4D Dynamic Scene Understanding

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Russell

Hilton

2019

Self Cite

We introduce the first approach to solve the challenging problem of unsupervised 4D visual scene understanding for complex dynamic scenes with multiple interacting people from multi-view video. Our approach simultaneously estimates a detailed model that includes a per-pixel semantically and temporally coherent reconstruction, together with instance-level segmentation exploiting photoconsistency, semantic and motion information. We further leverage recent advances in 3D pose estimation to constrain the joint semantic instance segmentation and 4D temporally coherent reconstruction. This enables per person semantic instance segmentation of multiple interacting people in complex dynamic scenes. Extensive evaluation of the joint visual scene understanding framework against state-of-theart methods on challenging indoor and outdoor sequences demonstrates a significant (≈ 40%) improvement in semantic segmentation, reconstruction and scene flow accuracy.