Optimal Choice of Motion Estimation Methods for Fine-Grained Action Classification with 3D Convolutional Networks

Abstract

Detecting and classifying human actions in videos is one of the current challenges in visual content analysis and mining. This paper presents a method for performing a finegrained classification of sport actions using a Siamese Spatio-Temporal Convolutional Neural Network (SSTCNN) model. This model takes RGB images and Optical Flow field as input data. Our first contribution is the comparison of different Optical flow methods and a study of their influence on the classification score. We also present different normalization methods for the optical flow that drastically impact results, boosting performances from 44% to 74% of accuracy. Our second contribution is the detection and classification of actions in videos performed using a sliding temporal window. It leads to a satisfying score of 81.3% over the whole dataset TTStroke-21.

Fine grained sport action recognition with Twin spatio-temporal convolutional neural networks

“…Our SSTCNN performs the best when considering all the classes but if the negative samples are not taken into account, our RGB model takes the lead. Furthermore in recent work [17], we show the effects of the Optical Flow normalization on the performance and recent results proves Siamese model to be more effective on the classification and detection task. Experimentation are still being conducted to understand and improve our results.…”

Three-Stream 3D/1D CNN for Fine-Grained Action Classification and Segmentation in Table Tennis

“…As presented in [16], the OF and its normalization can strongly impact the classification results. The same motion estimator reaching best classification performances is used thereafter.…”

“…The first modality is the raw information of pixel colour values. Motion is an important modality, extracted by optical flow, as investigated in [16]. It was proved to be efficient in terms of classification performance.…”

Two Stream Network for Stroke Detection in Table Tennis

“…As such, we decided to use optical flow as a modality to perform stroke detection. Inspired by [14], we decided to use DeepFlow method [23] to compute the optical flow from consecutive frames. The optical flow is computed from frames resized to 320 × 128.…”

“…Inspired from [8,14,15,21], this method combines the optical flow and features learned from the RGB stream in order to detect a stroke in table tennis and assess its duration. This implementation is an extension of the baseline code provided by the Sport Task organizers [11].…”