Content-aware broadcast soccer video retargeting using fuzzy logic

Abstract

A content-aware video retargeting method is proposed for playing broadcast soccer video in small displays. Four visual perception clues are predefined based on soccer game-specific knowledge and modelled by visual attention features firstly. Then, a fuzzy logic inference system is proposed to estimate visual attention values (AVs) of ball and players by fusing attention features. AVs are later used to determine the region of interest (ROI) of each frame. Finally, a retargeted video is generated by the ROI of each frame with polynomial curve fitting for temporal smoothing. Both subjective and objective evaluation results are promising.Introduction: An original video stream for TV or HDTV needs to be transformed into thumbnail videos for playing on small displays, such as a mobile phone. Early methods of direct down-sampling may bring viewers an uncomfortable watching experience [1], especially when small objects appear in original videos. Recent methods of video retargeting can be summarised into cropping based resizing [1, 2] and nonhomogeneous resizing [3,4]. Cropping based methods only remain in the region of interest (ROI). Non-homogeneous resizing brings distortion problems by allocating high resolution to important objects compared to non-important regions. Sports video is distortion-intolerant since the changing of spatial relationship among ball and players due to non-homogeneous resizing leads to a wrong understanding of sports events. It was also mentioned in [3,4]. As one of the most popular broadcast sports, soccer video is used in this Letter. In longview shots, objects (soccer ball and players) are too small to be recognised if the whole frame is shown on a small display. Therefore, we focus on soccer long-view shots and choose the cropping based method for retargeting. Normally, an ROI is selected according to a saliency map [3] to generate a resized video without considering video content. In [1], soccer ball and ball moving speed were used to determine the ROI. However, we find that a fast moving soccer ball is easily missed in the ROI by using the method in [1]. Besides soccer ball, players who have not been considered in [1], are also important clues to interesting content. In this Letter, we propose a content-aware cropping based retargeting for soccer long-view shots. The ROI is determined by four perception clues related to soccer ball and players, which are predefined based on soccer semantic constraints and domain-specific knowledge. The perception clues can be modified based on users' preference. Rule fusion is also a concern in this Letter. Three unique features are summarised as follows: 1. compared to [1], to achieve a content-aware retargeting, both soccer ball and players contribute to ROI detection; 2. visual perception is modelled by visual attention features; 3. fuzzy logic which collects human knowledge for decision making is introduced to fuse inference rules for ROI detection. The proposed method can be easily extended to other sports domains by applying domain-specific percepti...

Towards perceptual video cropping with curve fitting

“…(2) Seeking the optimal spatiotemporal path is typically very time-consuming or even intractable by considering too complex constraints. To this end, Gao et al [3] proposed a curve fitting method to accelerate the path seeking, but their method is designed only for soccer video.…”