Learning Crowdsourced User Preferences for Visual Summarization of Image Collections

Abstract

In this paper we propose a novel approach to selecting images suitable for inclusion in the visual summaries. The approach is grounded in insights about how people summarize image collections. We utilize the Amazon Mechanical Turk crowdsourcing platform to obtain a large number of manually created visual summaries as well as information about criteria for image inclusion in the summary. Based on these large-scale user tests, we propose an automatic image selection approach, which jointly utilizes the analysis of image content, context, popularity, visual aesthetic appeal as well as the sentiment derived from the comments posted on the images. In our approach we do not describe images based on their properties only, but also in the context of semantically related images, which improves robustness and effectively enables propagation of sentiment, aesthetic appeal as well as various inherent attributes associated with a particular group of images. We discuss the phenomenon of a low inter-user agreement, which makes an automatic evaluation of visual summaries a challenging task and propose a solution inspired by the text summarization and machine translation communities. The experiments performed on a collection of geo-referenced Flickr images demonstrate the effectiveness of our image selection approach.Index Terms-Crowdsourcing, image aesthetic appeal, image content and context, image set evaluation, learning to rank, sentiment analysis, social media, user-informed visual summarization.

Rethinking Summarization and Storytelling for Modern Social Multimedia

“…The problem of summarising image collections arises when there are e.g. large amounts of images from many users in a geographic area [12,13], or about a particular social event [14], or when it is necessary to generate a summarizing description (caption) [8]. Similarly, it is often needed to shorten or visualize long video sequences.…”

“…image search diversification [23][24][25][26][27]. Social multimedia summarization has further found its way in diverse applications ranging from personalized tweet summarization [28] to visual summarization of geographic areas and tourist routes [12,13,29,23] for POI recommendation and exploration. With the increased availability of a↵ordable wearables, in recent years lifelogging has started gaining popularity, where the goal is to generate a diary or a record of the day's activities and happenings by creating a summary or a story from the video/image data gathered [30,31].…”

“…Progress has been made in summarizing heterogeneous user-generated content with regards to relevance, representativeness, and diversity [15]. However, relevance criteria and their interplay may be much more complex than commonly assumed [12] and, in case of visual content, include additional factors such as content popularity, aesthetic appeal and sentiment. Thus we call for rethinking the foundations of summarization and storytelling.…”

“…Over the years many interesting metrics for evaluating (text) summaries were proposed, such as recall-oriented understudy for gisting evaluation (ROUGE) [37], bilingual evaluation understudy (BLEU) [38] and Pyramid Score [39]. Some of these were later on successfully adapted to the visual domain [40,12]. These initiatives had an impact on the progress in the field of summarization.…”

“…Although spontaneously captured, social multimedia may provide valuable insights about geographic city regions and their inhabitants. For example, user-generated content has been used to create summaries of geographic areas and tourist routes in location recommendation systems [13,12]. Sentiment extracted from social multimedia, in combination with neighborhood statistics was also proven invaluable for a more timely estimation of city livability and its causes [42].…”