Dense saliency-based spatiotemporal feature points for action recognition

Rapantzikos, Konstantinos; Avrithis, Yannis; Kollias, Stefanos

doi:10.1109/cvpr.2009.5206525

Cited by 118 publications

(58 citation statements)

References 15 publications

(16 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Willems et al propose a space-time detector based on the determinant of the 3D Hessian matrix, which is computationally efficient (use of integral videos) and is still on a par with current methods. Quite recently, Rapantzikos et al [9] proposed and evaluated the potential of spatiotemporal saliency to represent and detect actions in video sequences. Their method ranks among the best on standard human action datasets.…”

Section: Related Workmentioning

confidence: 99%

“…The competition is implemented through constrained minimization with the constraints being inspired by the Gestalt laws. From the mathematical point of view, the computational model we use is the same with the one we proposed in [9]. In this manuscript, we provide a detailed walkthrough of the model, highlight the main principles, and give a better insight of the method.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Spatiotemporal Features for Action Recognition and Salient Event Detection

2011

Self Cite

View full text Add to dashboard Cite

Although the mechanisms of human visual understanding remain partially unclear, computational models inspired by existing knowledge on human vision have emerged and applied to several fields. In this paper, we propose a novel method to compute visual saliency from video sequences by counting in the actual spatiotemporal nature of the video. The visual input is represented by a volume in space-time and decomposed into a set of feature volumes in multiple resolutions. Feature competition is used to produce a saliency distribution of the input implemented by constrained minimization. The proposed constraints are inspired by and associated with the Gestalt laws. There are a number of contributions in this approach, namely extending existing visual feature models to a volumetric representation, allowing competition across features, scales and voxels, and formulating constraints in accordance with perceptual principles. The resulting saliency volume is used to detect prominent spatiotemporal regions and consequently applied to action recognition and perceptually salient event detection in video sequences. Comparisons against established methods on public datasets are given and reveal the potential of the proposed model. The experiments include three action recognition scenarios and salient temporal segment detection in a movie database annotated by humans.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Spatiotemporal Features for Action Recognition and Salient Event Detection

2011

Self Cite

View full text Add to dashboard Cite

show abstract

“…To begin, we will start developing our theory for spatiotemporal scale selection with respect to the problem of detecting sparse spatio-temporal interest points [6,9,11,14,18,20,21,30,49,88,94,97,99,100,107,122,124,126,127], which may be regarded as a conceptually simplest problem domain because of the sparsity of spatio-temporal interest points and the close connection between this problem domain and the detection of spatial interest points for which there exists a theoretically well-founded and empirically tested framework regarding scale selection over the spatial domain [1,4,5,15,17,25,42,65,72,74,84,89,90,112]. Specifically, using a non-causal Gaussian spatio-temporal scale-space model, we will perform a theoretical analysis of the spatio-temporal scale selection properties of eight different types of spatiotemporal interest point detectors and show that seven of them: (i) the spatial Laplacian of the first-order temporal derivative, (ii) the spatial Laplacian of the second-order temporal derivative, (iii) the determinant of the spatial Hessian of the first-order temporal derivative, (iv) the determinant of the spatial Hessian of the second-order temporal derivative, (v) the determinant of the spatio-temporal Hessian matrix, (vi) the first-order temporal derivative of the determinant of the spatial Hessian matrix and (vii) the second-order temporal derivative of the determinant of the spatial Hessian matrix, do all lead to fully scale-covariant spatio-temporal scale estimates and scale-invariant feature responses under independent scaling transformations of the spatial and the temporal domains.…”

Section: Fig 4 the First-and Second-order Temporal Derivatives Of Thmentioning

confidence: 99%

Spatio-Temporal Scale Selection in Video Data

Lindeberg

2017

J Math Imaging Vis

View full text Add to dashboard Cite

This work presents a theory and methodology for simultaneous detection of local spatial and temporal scales in video data. The underlying idea is that if we process video data by spatio-temporal receptive fields at multiple spatial and temporal scales, we would like to generate hypotheses about the spatial extent and the temporal duration of the underlying spatio-temporal image structures that gave rise to the feature responses. For two types of spatio-temporal scale-space representations, (i) a non-causal Gaussian spatio-temporal scale space for offline analysis of pre-recorded video sequences and (ii) a time-causal and timerecursive spatio-temporal scale space for online analysis of real-time video streams, we express sufficient conditions for spatio-temporal feature detectors in terms of spatio-temporal receptive fields to deliver scale-covariant and scale-invariant feature responses. We present an in-depth theoretical analysis of the scale selection properties of eight types of spatio-temporal interest point detectors in terms of either: (i)-(ii) the spatial Laplacian applied to the first-and secondorder temporal derivatives, (iii)-(iv) the determinant of the spatial Hessian applied to the first-and second-order temporal derivatives, (v) the determinant of the spatio-temporal Hessian matrix, (vi) the spatio-temporal Laplacian and (vii)-(viii) the first-and second-order temporal derivatives of the determinant of the spatial Hessian matrix. It is shown that seven of these spatio-temporal feature detectors allow for provable scale covariance and scale invariance. Then, we describe a time-causal and time-recursive algorithm for detecting sparse spatio-temporal interest points from video streams and show that it leads to intuitively reasonable results. An experimental quantification of the accuracy of the spatio-temporal scale estimates and the amount of temporal delay obtained from these spatio-temporal interest point detectors is given, showing that: (i) the spatial and temporal scale selection properties predicted by the continuous theory are well preserved in the discrete implementation and (ii) the spatial Laplacian or the determinant of the spatial Hessian applied to the first-and second-order temporal derivatives leads to much shorter temporal delays in a timecausal implementation compared to the determinant of the spatio-temporal Hessian or the first-and second-order temporal derivatives of the determinant of the spatial Hessian matrix.

show abstract

“…The detector, called Cuboid, finds local maxima of a response function that contains a 2D Gaussian smoothing kernel and 1D temporal Gabor filters. Rapantzikos et al [112] used saliency to locate spatio-temporal points, where the saliency is computed by a global minimization process which leverages spatial proximity, scale, and feature similarity. To compute the feature similarity, they also utilized color, in addition to intensity and motion that are commonly adopted in other detectors.…”

Section: Spatio-temporal Local Featuresmentioning

confidence: 99%

High-level event recognition in unconstrained videos

Jiang

Bhattacharya

Chang

et al. 2012

Int J Multimed Info Retr

159

108

View full text Add to dashboard Cite

The goal of high-level event recognition is to automatically detect complex high-level events in a given video sequence. This is a difficult task especially when videos are captured under unconstrained conditions by nonprofessionals. Such videos depicting complex events have limited quality control, and therefore, may include severe camera motion, poor lighting, heavy background clutter, and occlusion. However, due to the fast growing popularity of such videos, especially on the Web, solutions to this problem are in high demands and have attracted great interest from researchers. In this paper, we review current technologies for complex event recognition in unconstrained videos. While the existing solutions vary, we identify common key modules and provide detailed descriptions along with some insights for each of them, including extraction and representation of low-level features across different modalities, classification strategies, fusion techniques, etc. Publicly available benchmark datasets, performance metrics, and related research forums are also described. Finally, we discuss promising directions for future research.

show abstract

Dense saliency-based spatiotemporal feature points for action recognition

Cited by 118 publications

References 15 publications

Spatiotemporal Features for Action Recognition and Salient Event Detection

Spatiotemporal Features for Action Recognition and Salient Event Detection

Spatio-Temporal Scale Selection in Video Data

High-level event recognition in unconstrained videos

Contact Info

Product

Resources

About