Evaluation and user studies with respect to video summarization and browsing

Abstract

The Informedia group at Carnegie Mellon University has since 1994 been developing and evaluating surrogates, summary interfaces, and visualizations for accessing digital video collections containing thousands of documents, millions of shots, and terabytes of data. This paper surveys the Informedia user studies that have taken place through the years, reporting on how these studies can provide a user pull complementing the technology push as automated video processing advances. The merits of discount usability techniques for iterative improvement and evaluation are presented, as well as the structure of formal empirical investigations with end users that have ecological validity while addressing the human computer interaction metrics of efficiency, effectiveness, and satisfaction. The difficulties in evaluating video summarization and browsing interfaces are discussed. Lessons learned from Informedia user studies are reported with respect to video summarization and browsing, ranging from the simplest portrayal of a single thumbnail to represent video stories, to collections of thumbnails in storyboards, to playable video skims, to video collages with multiple synchronized information perspectives.

Content-driven adaptation of on-line video

“…Extrinsic ones focus on the user expectative, that is, on the impact of the summary on the performance of a specific task (i.e., the user ability to answer questions about the original video content); they directly target the summarization goal, but they are application dependent. In this direction, some authors have also stressed and analyzed the influence of the end-user interface on the global summarization process [42,45].…”

Examining user interactions with video retrieval systems

“…One CMU expert had the same ENVIE interface as the analysts; the other two used a restricted "query-bybest-of-topic only" interface. The analysts do not have the same motivation or experience as these experts, so it is expected that they will not perform to the same levels as the system developers, just as students did not perform to the same levels as developers in Informedia user studies for TRECVID 2004 and 2005 interactive search tasks 1,11,15,16 . All four runs had available the ability to mark shots into two pools: those likely correct and relevant put into a "yes" pool in the shot collector pane shown to the extreme right of Figure 1, and those possibly relevant put into a "maybe" pool.…”

“…To bridge this semantic gap, the multimedia research community has invested in developing a Large-Scale Concept Ontology for Multimedia (LSCOM), whereby semantic concepts like "road" or "people" can be used for video retrieval 10 . These three access strategies, query-by-text, query-by-example, and query-by-concept, have been used by the Carnegie Mellon Informedia video search engine 1,11 and the MediaMill video search engine 8 for a number of years, with these systems scoring best for all of the TRECVID interactive video search evaluations since the task inception in 2002 4 . The user studies reported here fold in a specialized form of query-by-concept for the TRECVID topics: the ranked shot output of the fully automated search, which we label "query-by-best-of-topic" since this is a topic-specific shot list.…”

“…For the TRECVID 2005 and 2006 tasks, a "Shot Thumbnails" view (i.e., storyboard) is used exclusively to present results back to the user, as this view best addresses the need for shot-based retrieval based on visual characteristics embodied in the TRECVID search tasks. The ENVIE system was designed to provide much greater functionality than just shot-based retrieval support, however, with a great deal of foundation Informedia interface work going into the design and development of video browsing and summarization "views" of sets of video data 1,11,12 . The ENVIE system represents the metadata for a video set produced by a query action (e.g., query-by-text, query-by-example, query-byconcept, query-by-best-of-topic within a TRECVID context or query-by-geographic-map outside of TRECVID context) in XML, and presents the video set in a tab along with a "View Controller" manipulated by the user to determine how that XML is displayed in the interface.…”

“…These studies were surveyed in a 2006 paper reporting on how they can provide a user pull complementing the technology push as automated video processing advances 1 . The merits of discount usability techniques for iterative improvement and evaluation were presented in that same survey paper, as well as the structure of formal empirical investigations with end users that have ecological validity while addressing the human computer interaction metrics of efficiency, effectiveness, and satisfaction.…”