Fine grained sport action recognition with Twin spatio-temporal convolutional neural networks

Abstract

Human action recognition in video is one of the key problems in visual data interpretation. Despite intensive research, the recognition of actions with low inter-class variability remains a challenge. This paper presents a new Siamese Spatio-Temporal Convolutional Neural Network (SSTCNN) for this purpose. When applied to table tennis, it is possible to detect and recognize 20 table tennis strokes. The model has been trained on a specific dataset, so called TTStroke-21, recorded in natural conditions at the Faculty of Sports of the University of Bordeaux. Our model takes as inputs a RGB image sequence and its computed residual Optical Flow. The proposed siamese network architecture comprises 3 spatio-temporal convolutional layers, followed by a fully connected layer where data are fused. Our method reaches an accuracy of 91.4% against 43.1% for our baseline.

Application of deep learning in automatic detection of technical and tactical indicators of table tennis

“…Second, the table tennis ball moves fast; the low frame rate of the camera will blur the motion of the table tennis ball. Finally, in the rotation speed measurement, sufficient spherical information is often needed, which will cause the shooting camera to narrow the field of view and lose a lot of information [ 16 ].…”

A Deep Learning Approach for Table Tennis Forehand Stroke Evaluation System Using an IMU Sensor

“…The study's technical and practical aspects demonstrate that the proposed model has a high potential to be successfully applied in the second principal component of the Table Tennis shadow-play systems (see Figure 1 ). Unlike other similar studies that have used professional cameras [ 30 , 38 ] or multiple wearable sensors [ 31 , 40 ], the developed system only uses one object sensor to measure the Forehands' signals. The configuration and installation of vision-based sensing modality would cause the shadow-play assistance solution to be expensive, making it less affordable for the general population.…”

“…Dadashi et al [ 29 ] developed a swimming velocity estimation system with a single IMU to help coaches provide impressive guidance to trainees. Martin et al [ 30 ] presented a novel vision-based stroke classification system of Table Tennis with a new Twin Spatiotemporal CNN algorithm. Lim et al [ 31 ] developed a coaching assistant system of Table Tennis with three body-worn IMUs.…”

“…The applications developed by Zhang [ 38 ], Martin et al [ 30 ], Lim et al [ 31 ], Blank et al [ 39 ], and Liu et al [ 40 ] are the only related academically published papers in the area of Table Tennis. These studies, except Lim et al [ 31 ], only considered Table Tennis stroke classifications and did not consider the Table Tennis training issue.…”

“…These studies, except Lim et al [ 31 ], only considered Table Tennis stroke classifications and did not consider the Table Tennis training issue. Moreover, Zhang [ 38 ] and Martin et al [ 30 ] did not use any sensors as the sensing modalities; thus, their developed systems could be considerably expensive to implement in practical conditions. Both Lim et al [ 31 ] and Liu et al [ 40 ] developed body-worn sensor systems, but both designs were considerably complex for the players to set up and wear.…”