Spatio-Temporal Scale Selection in Video Data

Abstract

This work presents a theory and methodology for simultaneous detection of local spatial and temporal scales in video data. The underlying idea is that if we process video data by spatio-temporal receptive fields at multiple spatial and temporal scales, we would like to generate hypotheses about the spatial extent and the temporal duration of the underlying spatio-temporal image structures that gave rise to the feature responses. For two types of spatio-temporal scale-space representations, (i) a non-causal Gaussian spatio-temporal scale space for offline analysis of pre-recorded video sequences and (ii) a time-causal and timerecursive spatio-temporal scale space for online analysis of real-time video streams, we express sufficient conditions for spatio-temporal feature detectors in terms of spatio-temporal receptive fields to deliver scale-covariant and scale-invariant feature responses. We present an in-depth theoretical analysis of the scale selection properties of eight types of spatio-temporal interest point detectors in terms of either: (i)-(ii) the spatial Laplacian applied to the first-and secondorder temporal derivatives, (iii)-(iv) the determinant of the spatial Hessian applied to the first-and second-order temporal derivatives, (v) the determinant of the spatio-temporal Hessian matrix, (vi) the spatio-temporal Laplacian and (vii)-(viii) the first-and second-order temporal derivatives of the determinant of the spatial Hessian matrix. It is shown that seven of these spatio-temporal feature detectors allow for provable scale covariance and scale invariance. Then, we describe a time-causal and time-recursive algorithm for detecting sparse spatio-temporal interest points from video streams and show that it leads to intuitively reasonable results. An experimental quantification of the accuracy of the spatio-temporal scale estimates and the amount of temporal delay obtained from these spatio-temporal interest point detectors is given, showing that: (i) the spatial and temporal scale selection properties predicted by the continuous theory are well preserved in the discrete implementation and (ii) the spatial Laplacian or the determinant of the spatial Hessian applied to the first-and second-order temporal derivatives leads to much shorter temporal delays in a timecausal implementation compared to the determinant of the spatio-temporal Hessian or the first-and second-order temporal derivatives of the determinant of the spatial Hessian matrix.

Normative theory of visual receptive fields

“…When computing receptive field responses over multiple spatial and temporal scales, there is an issue about how the receptive field responses should be normalized so as to enable appropriate comparisons between receptive field responses at different scales. Issues of scale normalisation of the derivative based receptive fields defined from scalespace operations are treated in [36,58,59] regarding spatial receptive fields and in [21,38,55] regarding spatio-temporal receptive fields. a) Scale-normalized spatial receptive fields.…”

“…Several of these figures are similar to figures in [19] , [20] and in [21] . These results have, however, also been cleaned by replacing the previous time-causal models in [19] , [20] with the much better time-causal theory in [21] , and updating the distribution parameter c from the previous default value to the value found more suitable for computer vision algorithms that operate on these time-causal spatio-temporal receptive fields with real-time requirements of shorter temporal delays [38] .…”

Dense scale selection over space, time and space-time

“…of the scale-normalized spatio-temporal quasi quadrature entity (73) f (x, y, t) = g(x, y; s 0 ) g(t; τ 0 ) = 1 (2π) 3/2 s 0 √ τ 0 e −(x 2 +y 2 )/2s0 e −t 2 /2τ0 , for which the spatio-temporal scale-space representation is of the form (113) L(x, y, t; s 0 , τ 0 ) = g(x, y; s 0 + s) g(t; τ 0 + τ ), the spatial and temporal scale estimates will according to the theoretical analysis in (Lindeberg [44,45]) be given by ŝ = γ s s 0 2 − γ s s 0 , If we want to calibrate the spatial and temporal scale estimates such that the spatial and temporal scale estimates are equal to ŝ = s 0 and τ = τ 0 for a Gaussian blink of spatial extent s 0 and temporal duration τ 0 , we should therefore calibrate the phase compensated scale estimates ŝQ (x,y,t)L,comp and τQ (x,y,t)L,comp according to ŝQ the spatial scale estimate for a diffuse Gaussian edge will by combination of (106) with (120) be given by (123) ŝ = 2 + Γ s Γ s s 0 , whereas the temporal scale estimate for a Gaussian onset ramp will by combination of (107) with (121) be given by (124) τ = 1 + Γ τ Γ τ τ 0 .…”

“…Specifically, such scale estimates transform in a scale-covariant way under spatial scaling transformations of the image domain, which is a highly desirable property of a scale selection mechanism, since it implies that the scale estimates will automatically follow local scale variations in the image data. Corresponding local scale selection mechanisms can also be expressed over temporal and spatio-temporal domains (Lindeberg [46,44,45]).…”

Object Detection System to Help Navigating Visual Impairments

“…On the other hand, Geert Willems, Tinne Tuytelaars and Luc van Gool proposed "An efficient dense and scale-invariant spatiotemporal-temporal interest point detector" [12]. Meantime, Tony Lindeberg proposed "Spatio-temporal scale selection in video data" [13]. Also, I. Everts, J. van Gemert and T. Gevers proposed "Evaluation of color spatiotemporal interest points for human action recognition" [14].…”