Interactive video retrieval evaluation at a distance: comparing sixteen interactive video search systems in a remote setting at the 10th Video Browser Showdown

Abstract

The Video Browser Showdown addresses difficult video search challenges through an annual interactive evaluation campaign attracting research teams focusing on interactive video retrieval. The campaign aims to provide insights into the performance of participating interactive video retrieval systems, tested by selected search tasks on large video collections. For the first time in its ten year history, the Video Browser Showdown 2021 was organized in a fully remote setting and hosted a record number of sixteen scoring systems. In this paper, we describe the competition setting, tasks and results and give an overview of state-of-the-art methods used by the competing systems. By looking at query result logs provided by ten systems, we analyze differences in retrieval model performances and browsing times before a correct submission. Through advances in data gathering methodology and tools, we provide a comprehensive analysis of ad-hoc video search tasks, discuss results, task design and methodological challenges. We highlight that almost all top performing systems utilize some sort of joint embedding for text-image retrieval and enable specification of temporal context in queries for known-item search. Whereas a combination of these techniques drive the currently top performing systems, we identify several future challenges for interactive video search engines and the Video Browser Showdown competition itself.

vitrivr at the Lifelog Search Challenge 2022

“…One of the ways to evaluate research contributions in the area of retrieval systems is through interactive evaluation campaigns [17] such as the Lifelog Search Challenge (LSC) [5][6][7][8] or the Video Browser Showdown (VBS) [1, 12, 18-20, 25, 31], which have also been held entirely remotely in recent years [12,24] thanks to new and improved software for evaluation campaigns [26]. For the fifth edition of LSC, the dataset has increased in size, and now consists of 18 months of lifelog data [6].…”

Multimodal Interactive Lifelog Retrieval with vitrivr-VR

“…Co-located with the International Conference on Multimedia Retrieval (ICMR) the Lifelog Search Challenge (LSC) [3] is an annual competition for interactive lifelog retrieval systems. The LSC offers a platform for a wide variety of systems ranging from systems built for and exploring a specific aspect of lifelog retrieval to general purpose multimedia retrieval systems, and is modeled after the interactive evaluation campaign Video Browser Showdown (VBS) [5,8]. During the LSC, system operators must solve various information retrieval tasks as quickly as possible.…”

lifeXplore at the Lifelog Search Challenge 2022

“…The Lifelog Search Challenge (LSC) [2,3,5] facilitates the advancement on the topic of lifelog search and retrieval by offering a platform for international teams to compete against each other with their custom interactive systems. Drawing ideas from another popular competition for searching large video databases, known as the annual Video Browser Showdown (VBS) [6,10,12,14], the LSC is conducted as a live challenge, where competitors are tasked to search a known lifelog dataset [2,5] in order to find specific textually described moments. As the datasets are known to all teams beforehand, they can use every means at their disposal for data analysis, preprocessing and arrangement.…”