FREE: A Fast and Robust End-to-End Video Text Spotter

Abstract

Currently, video text spotting tasks usually fall into the four-staged pipeline: detecting text regions in individual images, recognizing localized text regions frame-wisely, tracking text streams and post-processing to generate final results. However, they may suffer from the huge computational cost as well as suboptimal results due to the interferences of low-quality text and the none-trainable pipeline strategy. In this paper, we propose a fast and robust end-to-end video text spotting framework named FREE by only recognizing the localized text stream onetime instead of frame-wise recognition. Specifically, FREE first employs a well-designed spatial-temporal detector that learns text locations among video frames. Then a novel text recommender is developed to select the highest-quality text from text streams for recognizing. Here, the recommender is implemented by assembling text tracking, quality scoring and recognition into a trainable module. It not only avoids the interferences from the low-quality text but also dramatically speeds up the video text spotting. FREE unites the detector and recommender into a whole framework, and helps achieve global optimization. Besides, we collect a large scale video text dataset for promoting the video text spotting community, containing 100 videos from 21 real-life scenarios. Extensive experiments on public benchmarks show our method greatly speeds up the text spotting process, and also achieves the remarkable state-of-the-art.

ICDAR 2021 Competition on Scene Video Text Spotting

“…- Annotations. The annotation strategy is same to LSVTD [4]. For each text region, the annotation items is as follows: (1) Polygon coordinate represents text location.…”

“…However, in many real-world applications, the sequence-level spotting results are the most urgently needed for users, while it's not what the user cares about for the framewise recognition results. Therefore, we propose the sequence-level evaluation protocals to evaluate the end-to-end performance, i.e., Recall s , Precision s , F-score s as used in [4]. Here, a predicted text sequence is regarded as a true postive if and only if it satisfies two constraints:…”

“…This is mainly due to: (1) Though 'Text in Videos' challenges have been recognized since 2013, the dataset is too small (containing only 49 videos from 7 different scenarios), which constrains the research on SVTS. (2) The lack of uniform evaluation metrics and benchmarks, as described in the literature [3,4]. For example, many methods only evaluate their localization performance on YVT and 'Text in Videos' [13,14], but few methods pay attention on the end-to-end evaluation.…”

“…-Inherited from LSVTD [3,4], the video text dataset is further extended, containing 129 video clips from 21 real-life scenarios. Compared to the existing ICDAR video text reading datasets, the extended dataset has some special features and challenges.…”

Video Text Tracking With a Spatio-Temporal Complementary Model

“…Yu et al [21] try to learn the feature embedding in an online association manner. Cheng et al [22] and [23] propose a tracker for the text instances by using the metric-learning method. For these tracking-by-detection approaches, the association step fully depends on the first step, i.e., detecting text in the spatial dimension.…”