Look, Read and Feel: Benchmarking Ads Understanding with Multimodal Multitask Learning

Abstract

Given the massive market of advertising and the sharply increasing online multimedia content (such as videos), it is now fashionable to promote advertisements (ads) together with the multimedia content. However, manually finding relevant ads to match the provided content is labor-intensive, and hence some automatic advertising techniques are developed. Since ads are usually hard to understand only according to its visual appearance due to the contained visual metaphor, some other modalities, such as the contained texts, should be exploited for understanding. To further improve user experience, it is necessary to understand both the ads' topic and sentiment. This motivates us to develop a novel deep multimodal multitask framework that integrates multiple modalities to achieve effective topic and sentiment prediction simultaneously for ads understanding. In particular, in our framework termed Deep 2 Ad, we first extract multimodal information from ads and learn high-level and comparable representations. The visual metaphor of the ad is decoded in an unsupervised manner. The obtained representations are then fed into the proposed hierarchical multimodal attention modules to learn task-specific representations for final prediction. A multitask loss function is also designed to jointly train both the topic and sentiment prediction models in an end-to-end manner, where bottom-layer parameters are shared to alleviate over-fitting. We conduct extensive experiments on a large-scale advertisement dataset and achieve state-of-the-art performance for both prediction tasks. The obtained results could be utilized as a benchmark for ads understanding.

Exploring the opinions of the YouTube visitors towards advertisements and its influence on purchase intention among viewers

“…In a similar fashion, participants in the current study have explained that some advertisements they had exposed to were creative and enjoyable. Previous studies reported that creative advertisement content was thought-provoking (Vedula, et al, 2017;Zhang et al, 2019). YouTube presents moving images to a wide audience and has established itself as one of the most successful and most visited online video-sharing services (Snelson, 2011).…”

Affect2MM: Affective Analysis of Multimedia Content Using Emotion Causality

“…Chen et al [12], Ali et al [1], Wei et al [74] have performed affective multiclass classification on images collected from popular websites. Pilli et al [57], Hussain et al [26], and Zhang et al [82] studied predicting sentiments in image advertisements. Vedula et al [70] extended this idea and developed an advertisement recommendation system using sentiments in advertisement content.…”

Multimodal learning for quality of experience in video-to-retail applications: algorithms and deployment

“…Therefore, many applications have been developed to support this combination. These applications include fashion products retrieval from videos [30], contextual ads insertion [45,162], etc. We term them Video-to-Retail (V2R) applications.…”