Considering Human Perception and Memory in Interactive Multimedia Retrieval Evaluations

Abstract

Experimental evaluations dealing with visual known-item search tasks, where real users look for previously observed and memorized scenes in a given video collection, represent a challenging methodological problem. Playing a searched "known" scene to users prior to the task start may not be sufficient in terms of scene memorization for re-identification (i.e., the search need may not necessarily be successfully "implanted"). On the other hand, enabling users to observe a known scene played in a loop may lead to unrealistic situations where users can exploit very specific details that would not remain in their memory in a common case. To address these issues, we present a proof-of-concept implementation of a new visual known-item search task presentation methodology that relies on a recently introduced deep saliency estimation method to limit the amount of revealed visual video contents. A filtering process predicts and subsequently removes information which in an unconstrained setting would likely not leave a lasting impression in the memory of a human observer. The proposed presentation setting is compliant with a realistic assumption that users perceive and memorize only a limited amount of information, and at the same time allows to play the known scene in the loop for verification purposes. The new setting also serves as a search clue equalizer, limiting the rich set of present exploitable content features in video and thus unifies the perceived information by different users. The performed evaluation demonstrates the feasibility of such a task presentation by showing that retrieval is still possible based on query videos processed by the proposed method. We postulate that such information incomplete tasks constitute the necessary next step to challenge and assess interactive multimedia retrieval systems participating at visual known-item search evaluation campaigns.

Lightweight sensing-computing-decision collaboration enhancement for multi-mobile terminals

“…在探 讨终端设备上分布式多模态数据流的感知计算融合时, 我们不可避免地面临三个核心挑战. 首先, 终 端设备感知配置(包括分辨率、采样率、传输延迟等参数)的动态变化, 往往会引发无效信息对带宽 的占用, 进而削弱模型的分析精确度 [27,28] . 同时, 有研究工作 [29,30] 指出, 由于现实环境的不断演变, 静态的最优配置难以持续适应感知数据的动态变化.…”

A Task Category Space for User-Centric Comparative Multimedia Search Evaluations

“…While there have been some early experiments in this direction [23], it is largely unclear how such effects could be simulated realistically and further research on query representation methods modeling human visual memory is needed.…”

Is the Reign of Interactive Search Eternal? Findings from the Video Browser Showdown 2020

“…Even though video distortion of the played scene was suggested before, at the competition, the 20s clips were played without any blurring. The main reason was that effects of query distortion are not yet well understood and need a further study [51]. For textual KIS tasks, only a textual description for a 20-second target segment was presented, which additionally required the teams to imagine how the scene could look like.…”