Crowdsourced time-sync video tagging using temporal and personalized topic modeling

Abstract

Time-sync video tagging aims to automatically generate tags for each video shot. It can improve the user's experience in previewing a video's timeline structure compared to traditional schemes that tag an entire video clip. In this paper, we propose a new application which extracts time-sync video tags by automatically exploiting crowdsourced comments from video websites such as Nico Nico Douga, where videos are commented on by online crowd users in a time-sync manner. The challenge of the proposed application is that users with bias interact with one another frequently and bring noise into the data, while the comments are too sparse to compensate for the noise. Previous techniques are unable to handle this task well as they consider video semantics independently, which may overfit the sparse comments in each shot and thus fail to provide accurate modeling. To resolve these issues, we propose a novel temporal and personalized topic model that jointly considers temporal dependencies between video semantics, users' interaction in commenting, and users' preferences as prior knowledge. Our proposed model shares knowledge across video shots via users to enrich the short comments, and peels off user interaction and user bias to solve the noisy-comment problem. Log-likelihood analyses and user studies on large datasets show that the proposed model outperforms several state-of-the-art baselines in video tagging quality. Case studies also demonstrate our model's capability of extracting tags from the crowdsourced short and noisy comments.

Crowdsourcing Fundamentals: Definition and Typology

“…The crowd may be rewarded monetarily, with prizes, with extra talk time or some other form of recognition. In some cases, the reward could be just intellectual satisfaction" (continued) Wu et al (2014) 728 "a process that involves outsourcing tasks to a distributed group of people, which is normally much cheaper than hiring experts" 10. Zeinalipour-Yazti et al (2013) 1240 "a distributed problem solving model where a population of undefined size, engages in the solution of a complex problem for monetary or ethical (i.e., intellectual satisfaction) benefit through an open call" 11.…”

A collaborative approach for semantic time-based video annotation using gamification

“…The system works by finding similar posts, filtering and re-ranking results based on tag co-occurrence and frequency, improving the suggestions on tagged posts. Wu et al [21] follows a similar concept by presenting an application that extracts time-sync video tags by exploiting crowdsourced comments from video websites.…”

Video Highlight Shot Extraction with Time-Sync Comment

“…[13] first use TSC to tag selected video shots through topic models. His work mainly focus on how to annotate a given shot by learning existing ones.…”

“…Nico Nico Douga 1 in Japan and bilibili 2 in China are two typical examples. Videos on these sites consist of external textual information known as time-sync comments (or simply TSC, the term first introduced in [13]) or barrages (in [7]). An example of TSC 3 is shown in Figure 1.…”

“…To the best of our knowledge, limited work has been done with the research on TSC [7,13,14], let alone its application on video extraction. In this paper, we aim to extract the latent video highlight shots that are similar to the given video with the help of TSC.…”