A Framework for Effective Known-item Search in Video

Lokoč, Jakub; Kovalčík, Gregor; Souček, Tomáš; Moravec, Jaroslav; Čech, Přemysl

doi:10.1145/3343031.3351046

Cited by 36 publications

(32 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The videos are automatically filtered to remove short videos and videos with a resolution below 480p. For longer videos, as in AVA, we use [39] to extract scenes without shot changes. In addition, we manually reviewed each sampled video to ensure it is high quality: i.e., we removed grainy videos as well as videos with excessive camera motion or shot changes.…”

Section: Video Selectionmentioning

confidence: 99%

TAO: A Large-Scale Benchmark for Tracking Any Object

Dave

Khurana

Tokmakov

et al. 2020

Lecture Notes in Computer Science

103

View full text Add to dashboard Cite

For many years, multi-object tracking benchmarks have focused on a handful of categories. Motivated primarily by surveillance and self-driving applications, these datasets provide tracks for people, vehicles, and animals, ignoring the vast majority of objects in the world. By contrast, in the related field of object detection, the introduction of large-scale, diverse datasets (e.g., COCO) have fostered significant progress in developing highly robust solutions. To bridge this gap, we introduce a similarly diverse dataset for Tracking Any Object (TAO) 4. It consists of 2,907 high resolution videos, captured in diverse environments, which are half a minute long on average. Importantly, we adopt a bottomup approach for discovering a large vocabulary of 833 categories, an order of magnitude more than prior tracking benchmarks. To this end, we ask annotators to label objects that move at any point in the video, and give names to them post factum. Our vocabulary is both significantly larger and qualitatively different from existing tracking datasets. To ensure scalability of annotation, we employ a federated approach that focuses manual effort on labeling tracks for those relevant objects in a video (e.g., those that move). We perform an extensive evaluation of state-ofthe-art trackers and make a number of important discoveries regarding large-vocabulary tracking in an open-world. In particular, we show that existing single-and multi-object trackers struggle when applied to this scenario in the wild, and that detection-based, multi-object trackers are in fact competitive with user-initialized ones. We hope that our dataset and analysis will boost further progress in the tracking community.

show abstract

Section: Video Selectionmentioning

confidence: 99%

TAO: A Large-Scale Benchmark for Tracking Any Object

Dave

Khurana

Tokmakov

et al. 2020

Lecture Notes in Computer Science

103

View full text Add to dashboard Cite

show abstract

“…However, a survey of possible systems is beyond the scope of this paper, and so we summarize basic features of just several selected recent systems that performed competitively at evaluation campaigns. Furthermore, two successful systems VIRET [26] and SOMHunter [14] are both described in more detail in Section 4.2.…”

Section: Interactive Search Systemsmentioning

confidence: 99%

“…However, real search use-cases differ from the laboratory evaluation setting and so the observed performance could differ too. For example, it might happen that real users overlook the searched item in the result list due to too many items on a display page [26]. On the other hand, if first pages of the result list are not satisfactory, users can try to reformulate the query to localize searched items with a different set of keywords and still solve a task in a given time limit.…”

Section: Interactive Search With W2vv++mentioning

confidence: 99%

“…Two different interactive search prototypes were tested for a set of KIS tasks at VBS 2020 -SOMHunter [14] and VIRET [26]. Both of them use the same set I of selected representative video frames (sorted for each video by time) and provide a text search mode for the frames using the selected W2VV++ variant with an option to formulate a temporal query [26]. For the similarity of two images, the prototypes used the same descriptor universe R 128 designed for the W2VV++ model.…”

Section: Tested Interactive Search Prototypesmentioning

confidence: 99%

See 1 more Smart Citation

A W2VV++ Case Study with Automated and Interactive Text-to-Video Retrieval

Lokoč

Souček

Veselý

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

As reported by respected evaluation campaigns focusing both on automated and interactive video search approaches, deep learning started to dominate the video retrieval area. However, the results are still not satisfactory for many types of search tasks focusing on high recall. To report on this challenging problem, we present two orthogonal task-based performance studies centered around the state-of-the-art W2VV++ query representation learning model for video retrieval. First, an ablation study is presented to investigate which components of the model are effective in two types of benchmark tasks focusing on high recall. Second, interactive search scenarios from the Video Browser Showdown are analyzed for two winning prototype systems implementing a selected variant of the model and providing additional querying and visualization components. The analysis of collected logs demonstrates that even with the state-of-the-art text search video retrieval model, it is still auspicious to integrate users into the search process for task types, where high recall is essential. CCS CONCEPTS • Information systems → Video search.

show abstract

“…The relevance score of frame is computed as exp(− • cos ( , )) for some arbitrarily chosen steepness . Possible temporal combinations of multiple queries are implemented using a simple heuristic model [10].…”

Section: Content-based Search Modelmentioning

confidence: 99%

SOMHunter: Lightweight Video Search System with SOM-Guided Relevance Feedback

Kratochvíl

Mejzlík

Veselý

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

In the last decade, the Video Browser Showdown (VBS) became a comparative platform for various interactive video search tools competing in selected video retrieval tasks. However, the participation of new teams with an own, novel tool is prohibitively timedemanding because of the large number and complexity of components required for constructing a video search system from scratch. To partially alleviate this difficulty, we provide an open-source version of the lightweight known-item search system SOMHunter that competed successfully at VBS 2020. The system combines several features for text-based search initialization and browsing of large result sets; in particular a variant of W2VV++ model for text search, temporal queries for targeting sequences of frames, several types of displays including the eponymous self-organizing map view, and a feedback-based approach for maintaining the relevance scores inspired by PICHunter. The minimalistic, easily extensible implementation of SOMHunter should serve as a solid basis for constructing new search systems, thus facilitating easier exploration of new video retrieval ideas. CCS CONCEPTS • Information systems → Video search.

show abstract

A Framework for Effective Known-item Search in Video

Cited by 36 publications

References 11 publications

TAO: A Large-Scale Benchmark for Tracking Any Object

TAO: A Large-Scale Benchmark for Tracking Any Object

A W2VV++ Case Study with Automated and Interactive Text-to-Video Retrieval

SOMHunter: Lightweight Video Search System with SOM-Guided Relevance Feedback

Contact Info

Product

Resources

About