Time‐based tags for fiction movies: comparing experts to novices using a video labeling game

Abstract

The cultural heritage sector has embraced social tagging as a way to increase both access to online content and to engage users with their digital collections. In this article, we build on two current lines of research. (a) We use Waisda?, an existing labeling game, to add timebased annotations to content. (b) In this context, we investigate the role of experts in human-based computation (nichesourcing). We report on a small-scale experiment in which we applied Waisda? to content from film archives. We study the differences in the type of timebased tags between experts and novices for film clips in a crowdsourcing setting. The findings show high similarity in the number and type of tags (mostly factual). In the less frequent tags, however, experts used more domain-specific terms. We conclude that competitive games are not suited to elicit real expert-level descriptions. We also confirm that providing guidelines, based on conceptual frameworks that are more suited to moving images in a time-based fashion, could result in increasing the quality of the tags, thus allowing for creating more tag-based innovative services for online audiovisual heritage.

Socio-Economic Diversity in Human Annotations

“…Discussion. Our findings are consistent with previous studies using the same classification method [12,15,19,23]. To begin, the majority of crowdsourced descriptions are at the conceptual level.…”

“…We also have consistent finding with Estrada et al [12], in that the third most frequently used annotation is abstract/What (e.g., "neatness"), as people use abstract terms to describe events or actions in a scene, such as emotions and judgment. Abstract annotations are subjective in nature and are used to ascertain what other users think about a video [23].…”

“…They discovered that television production experts with extensive domain knowledge do not compensate for a lack of indexing training and motivation. On the other hand, cost and scalability constraints limit the amount of expert annotation that can be collected, and experts may also have a different understanding of video content than the general public [12]. Automatic annotation solves the problem of scalability but is restrained by the types of available annotations.…”

“…There are few studies on the diversity of video annotations generated by crowd workers. Some studies have explored how to use video labeling games to annotate movie clips [12] and TV shows [15,19], but the number of annotations they generate and the number of video clips they use are small. More comparisons between our study and these past studies are presented in Section 4.…”

Moving Image User-Generated Description

“…Since the users were not divided based on domain expertise, the tags should be relatively similar in nature-as the data confirm. Previous studies recommended specific tag types such as cinematography, emotions, explanations, and facts in their instructions to users (Estrada et al, 2016). This study did not include recommendations, instead the instructions stated, "A tag, if you are unfamiliar with tagging, should provide some description of the video that would help yourself and/or others find it through search or browsing online."…”