Coding Images with Local Features

Dickscheid, Timo; Schindler, Frank; Förstner, Wolfgang

doi:10.1007/s11263-010-0340-z

Cited by 45 publications

(51 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To begin, we will start developing our theory for spatiotemporal scale selection with respect to the problem of detecting sparse spatio-temporal interest points [6,9,11,14,18,20,21,30,49,88,94,97,99,100,107,122,124,126,127], which may be regarded as a conceptually simplest problem domain because of the sparsity of spatio-temporal interest points and the close connection between this problem domain and the detection of spatial interest points for which there exists a theoretically well-founded and empirically tested framework regarding scale selection over the spatial domain [1,4,5,15,17,25,42,65,72,74,84,89,90,112]. Specifically, using a non-causal Gaussian spatio-temporal scale-space model, we will perform a theoretical analysis of the spatio-temporal scale selection properties of eight different types of spatiotemporal interest point detectors and show that seven of them: (i) the spatial Laplacian of the first-order temporal derivative, (ii) the spatial Laplacian of the second-order temporal derivative, (iii) the determinant of the spatial Hessian of the first-order temporal derivative, (iv) the determinant of the spatial Hessian of the second-order temporal derivative, (v) the determinant of the spatio-temporal Hessian matrix, (vi) the first-order temporal derivative of the determinant of the spatial Hessian matrix and (vii) the second-order temporal derivative of the determinant of the spatial Hessian matrix, do all lead to fully scale-covariant spatio-temporal scale estimates and scale-invariant feature responses under independent scaling transformations of the spatial and the temporal domains.…”

Section: Fig 4 the First-and Second-order Temporal Derivatives Of Thmentioning

confidence: 99%

Spatio-Temporal Scale Selection in Video Data

Lindeberg

2017

J Math Imaging Vis

View full text Add to dashboard Cite

This work presents a theory and methodology for simultaneous detection of local spatial and temporal scales in video data. The underlying idea is that if we process video data by spatio-temporal receptive fields at multiple spatial and temporal scales, we would like to generate hypotheses about the spatial extent and the temporal duration of the underlying spatio-temporal image structures that gave rise to the feature responses. For two types of spatio-temporal scale-space representations, (i) a non-causal Gaussian spatio-temporal scale space for offline analysis of pre-recorded video sequences and (ii) a time-causal and timerecursive spatio-temporal scale space for online analysis of real-time video streams, we express sufficient conditions for spatio-temporal feature detectors in terms of spatio-temporal receptive fields to deliver scale-covariant and scale-invariant feature responses. We present an in-depth theoretical analysis of the scale selection properties of eight types of spatio-temporal interest point detectors in terms of either: (i)-(ii) the spatial Laplacian applied to the first-and secondorder temporal derivatives, (iii)-(iv) the determinant of the spatial Hessian applied to the first-and second-order temporal derivatives, (v) the determinant of the spatio-temporal Hessian matrix, (vi) the spatio-temporal Laplacian and (vii)-(viii) the first-and second-order temporal derivatives of the determinant of the spatial Hessian matrix. It is shown that seven of these spatio-temporal feature detectors allow for provable scale covariance and scale invariance. Then, we describe a time-causal and time-recursive algorithm for detecting sparse spatio-temporal interest points from video streams and show that it leads to intuitively reasonable results. An experimental quantification of the accuracy of the spatio-temporal scale estimates and the amount of temporal delay obtained from these spatio-temporal interest point detectors is given, showing that: (i) the spatial and temporal scale selection properties predicted by the continuous theory are well preserved in the discrete implementation and (ii) the spatial Laplacian or the determinant of the spatial Hessian applied to the first-and second-order temporal derivatives leads to much shorter temporal delays in a timecausal implementation compared to the determinant of the spatio-temporal Hessian or the first-and second-order temporal derivatives of the determinant of the spatial Hessian matrix.

show abstract

Section: Fig 4 the First-and Second-order Temporal Derivatives Of Thmentioning

confidence: 99%

Spatio-Temporal Scale Selection in Video Data

Lindeberg

2017

J Math Imaging Vis

View full text Add to dashboard Cite

show abstract

“…For the results shown in this paper, the SIFT detector is used for newly appearing features and provides sufficiently distributed points. For sequences with very low texture content, a combination of different scale invariant feature detectors should be considered [21,22,23]. For the tracking from frame to frame, the KLT tracker provides higher accuracy and less outliers than feature matching techniques.…”

Section: Feature Detection and Trackingmentioning

confidence: 99%

Foreground Segmentation from Occlusions Using Structure and Motion Recovery

Cordes

Scheuermann

Rosenhahn

et al. 2013

Communications in Computer and Information Science

View full text Add to dashboard Cite

Abstract. The segmentation of foreground objects in camera images is a fundamental step in many computer vision applications. For visual effect creation, the foreground segmentation is required for the integration of virtual objects between scene elements. On the other hand, camera and scene estimation is needed to integrate the objects perspectively correct into the video. In this paper, discontinued feature tracks are used to detect occlusions. If these features reappear after their occlusion, they are connected to the correct previously discontinued trajectory during sequential camera and scene estimation. The combination of optical flow for features in consecutive frames and SIFT matching for the wide baseline feature connection provides accurate and stable feature tracking. The knowledge of occluded parts of a connected feature track is used to feed an efficient segmentation algorithm which crops the foreground image regions automatically. The presented graph cut based segmentation uses a graph contraction technique to minimize the computational expense. The presented application in the integration of virtual objects into video. For this application, the accurate estimation of camera and scene is crucial. The segmentation is used for the automatic occlusion of the integrated objects with foreground scene content. Demonstrations show very realistic results.

show abstract

“…For the results shown in this paper, the SIFT detector is used for newly appearing features and provides sufficiently distributed points. For sequences with very low texture content, a combination of different scale invariant feature detectors may be considered (Lowe, 2004;Matas et al, 2002;Dickscheid et al, 2010). For the tracking from frame to frame, the KLT tracker provides higher accuracy and less outliers than feature matching techniques.…”

Section: Feature Detection and Trackingmentioning

confidence: 99%

Occlusion Handling for the Integration of Virtual Objects Into Video

Cordes

Scheuermann

Rosenhahn

et al. 2012

Proceedings of the International Conference on Computer Vision Theory and Applications

View full text Add to dashboard Cite

Abstract:This paper demonstrates how to effectively exploit occlusion and reappearance information of feature points in structure and motion recovery from video. Due to temporary occlusion with foreground objects, feature tracks discontinue. If these features reappear after their occlusion, they are connected to the correct previously discontinued trajectory during sequential camera and scene estimation. The combination of optical flow for features in consecutive frames and SIFT matching for the wide baseline feature connection provides accurate and stable feature tracking. The knowledge of occluded parts of a connected feature track is used to feed a segmentation algorithm which crops the foreground image regions automatically. The resulting segmentation provides an important step in scene understanding which eases integration of virtual objects into video significantly. The presented approach enables the automatic occlusion of integrated virtual objects with foreground regions of the video. Demonstrations show very realistic results in augmented reality.

show abstract

Coding Images with Local Features

Cited by 45 publications

References 40 publications

Spatio-Temporal Scale Selection in Video Data

Spatio-Temporal Scale Selection in Video Data

Foreground Segmentation from Occlusions Using Structure and Motion Recovery

Occlusion Handling for the Integration of Virtual Objects Into Video

Contact Info

Product

Resources

About