Detection of Important Scenes in Baseball Videos via a Time-Lag-Aware Multimodal Variational Autoencoder

Abstract

A new method for the detection of important scenes in baseball videos via a time-lag-aware multimodal variational autoencoder (Tl-MVAE) is presented in this paper. Tl-MVAE estimates latent features calculated from tweet, video, and audio features extracted from tweets and videos. Then, important scenes are detected by estimating the probability of the scene being important from estimated latent features. It should be noted that there exist time-lags between tweets posted by users and videos. To consider the time-lags between tweet features and other features calculated from corresponding multiple previous events, the feature transformation based on feature correlation considering such time-lags is newly introduced to the encoder in MVAE in the proposed method. This is the biggest contribution of the Tl-MVAE. Experimental results obtained from actual baseball videos and their corresponding tweets show the effectiveness of the proposed method.

Time-Lag Aware Latent Variable Model for Prediction of Important Scenes Using Baseball Videos and Tweets

“…The encoder, decoder, and important scene predictor consisted of multiple fully connected layers, respectively. Unlike the conventional method [ 14 ] which trains a part of the encoder considering time lags, Tl-LVM can train the entire network considering time lags based on the new loss function. Therefore, this figure does not represent a time-lag aware transformation in the encoder as in the figure of the conventional method.…”

“…The method in [ 22 ] uses a multimodal variational autoencoder (MVAE) [ 24 ] using tweets and visual information as the latent variable model (LVM) to detect fake news. The effectiveness of using LVM to consider both videos and tweets has also been reported in [ 14 , 16 ]. The MVAE can discover correlations between modalities by learning shared representations.…”

“…Twitter allows users to receive and post short text messages called tweets . Thus, using features extracted from both videos and tweets posted by viewers has improved the prediction and detection performance of important scenes in sports videos [ 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 ].…”

“…For constructing MVAE-based methods using tweets and videos, the following problem is considered: viewers watching baseball videos post tweets inspired by several previous events, such as an RBI hit and home run. Thus, since tweets and multiple corresponding events have time lags, the conventional methods [ 14 , 16 ] detect important scenes through the consideration of tweets and events. Specifically, the conventional method [ 14 ] detects important scenes via a time-lag-aware multimodal variational autoencoder that has the encoder considering the time lags between tweets and events, and the conventional method [ 16 ] detects them based on a generative adversarial network-based approach using features transformed via bidirectional time-lag aware deep multiset canonical correlation analysis.…”

“…Thus, since tweets and multiple corresponding events have time lags, the conventional methods [ 14 , 16 ] detect important scenes through the consideration of tweets and events. Specifically, the conventional method [ 14 ] detects important scenes via a time-lag-aware multimodal variational autoencoder that has the encoder considering the time lags between tweets and events, and the conventional method [ 16 ] detects them based on a generative adversarial network-based approach using features transformed via bidirectional time-lag aware deep multiset canonical correlation analysis. However, these methods cannot simultaneously train a time-lag aware feature transformation network and network for the detection of important scenes.…”