Automatic Understanding of Image and Video Advertisements

Hussain, Zaeem; Zhang, Mingda; Zhang, Xiaozhong; Ye, Keren; Thomas, Christopher; Agha, Zuha; Ong, Nathan; Kovashka, Adriana

doi:10.1109/cvpr.2017.123

Cited by 131 publications

(174 citation statements)

References 70 publications

Supporting

Mentioning

164

Contrasting

Order By: Relevance

“…The creatives dataset [1] is one of the key enablers of the proposed recommender system. This dataset was introduced in [15], where the authors focused on automatically understanding the content in ad images and videos from a computer vision perspective. The dataset has ad creatives with annotations including topic (category), questions and answers (e.g., reasoning behind the ad, expected user response due the ad).…”

Section: Automatic Understanding Of Ad Creativesmentioning

confidence: 99%

“…A key enabler in pursuing the above data driven approach for inferring themes is that of a dataset of ad creatives spanning multiple advertisers. Such a dataset [1] spanning 64, 000 ad images was recently introduced in [15], and also used in the followup work [30]. The collective focus in the above works [15,30] was on understanding ad creatives in terms of sentiment, symbolic references and VQA.…”

Section: Introductionmentioning

confidence: 99%

“…Such a dataset [1] spanning 64, 000 ad images was recently introduced in [15], and also used in the followup work [30]. The collective focus in the above works [15,30] was on understanding ad creatives in terms of sentiment, symbolic references and VQA. In particular, no connection was made with the brands inferred in creatives, and the associated world knowledge on the inferred brands.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Recommending Themes for Ad Creative Design via Visual-Linguistic Representations

Zhou

Mishra

Verma

et al. 2020

Proceedings of the Web Conference 2020

View full text Add to dashboard Cite

There is a perennial need in the online advertising industry to refresh ad creatives, i.e., images and text used for enticing online users towards a brand. Such refreshes are required to reduce the likelihood of ad fatigue among online users, and to incorporate insights from other successful campaigns in related product categories. Given a brand, to come up with themes for a new ad is a painstaking and time consuming process for creative strategists. Strategists typically draw inspiration from the images and text used for past ad campaigns, as well as world knowledge on the brands. To automatically infer ad themes via such multimodal sources of information in past ad campaigns, we propose a theme (keyphrase) recommender system for ad creative strategists. The theme recommender is based on aggregating results from a visual question answering (VQA) task, which ingests the following: (i) ad images, (ii) text associated with the ads as well as Wikipedia pages on the brands in the ads, and (iii) questions around the ad. We leverage transformer based cross-modality encoders to train visual-linguistic representations for our VQA task. We study two formulations for the VQA task along the lines of classification and ranking; via experiments on a public dataset, we show that cross-modal representations lead to significantly better classification accuracy and ranking precision-recall metrics. Cross-modal representations show better performance compared to separate image and text representations. In addition, the use of multimodal information shows a significant lift over using only textual or visual information.

show abstract

Section: Automatic Understanding Of Ad Creativesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Recommending Themes for Ad Creative Design via Visual-Linguistic Representations

Zhou

Mishra

Verma

et al. 2020

Proceedings of the Web Conference 2020

View full text Add to dashboard Cite

show abstract

“…Figure 1: An example of a complex message portrayed by an image-text pair elucidating the semantic gap between the textual information and the image content. (Source: [18])…”

Section: Introductionmentioning

confidence: 99%

Understanding, Categorizing and Predicting Semantic Image-Text Relations

Otto

Springstein

Anand

et al. 2019

Proceedings of the 2019 on International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

Two modalities are often used to convey information in a complementary and beneficial manner, e.g., in online news, videos, educational resources, or scientific publications. The automatic understanding of semantic correlations between text and associated images as well as their interplay has a great potential for enhanced multimodal web search and recommender systems. However, automatic understanding of multimodal information is still an unsolved research problem. Recent approaches such as image captioning focus on precisely describing visual content and translating it to text, but typically address neither semantic interpretations nor the specific role or purpose of an image-text constellation. In this paper, we go beyond previous work and investigate, inspired by research in visual communication, useful semantic image-text relations for multimodal information retrieval. We derive a categorization of eight semantic image-text classes (e.g., "illustration" or "anchorage") and show how they can systematically be characterized by a set of three metrics: cross-modal mutual information, semantic correlation, and the status relation of image and text. Furthermore, we present a deep learning system to predict these classes by utilizing multimodal embeddings. To obtain a sufficiently large amount of training data, we have automatically collected and augmented data from a variety of data sets and web resources, which enables future research on this topic. Experimental results on a demanding test set demonstrate the feasibility of the approach.

show abstract

“…Most of the existing works in the literature [3,4] involve identification of logos in video frames, or detecting advertise-The ADAPT Centre for Digital Content Technology is funded under the SFI Research Centres Programme (Grant 13/RC/2106) and is co-funded under the European Regional Development Fund.…”

Section: Introductionmentioning

confidence: 99%

Localizing Adverts in Outdoor Scenes

Dev

Hossari

Nicholson

et al. 2019

2019 IEEE International Conference on Multimedia &Amp; Expo Workshops (ICMEW)

View full text Add to dashboard Cite

Online videos have witnessed an unprecedented growth over the last decade, owing to wide range of content creation. This provides the advertisement and marketing agencies plethora of opportunities for targeted advertisements. Such techniques involve replacing an existing advertisement in a video frame, with a new advertisement. However, such post-processing of online videos is mostly done manually by video editors. This is cumbersome and time-consuming. In this paper, we propose DeepAds -a deep neural network, based on the simple encoder-decoder architecture, that can accurately localize the position of an advert in a video frame. Our approach of localizing billboards in outdoor scenes using neural nets, is the first of its kind, and achieves the best performance. We benchmark our proposed method with other semantic segmentation algorithms, on a public dataset of outdoor scenes with manually annotated billboard binary maps.

show abstract

Automatic Understanding of Image and Video Advertisements

Cited by 131 publications

References 70 publications

Recommending Themes for Ad Creative Design via Visual-Linguistic Representations

Recommending Themes for Ad Creative Design via Visual-Linguistic Representations

Understanding, Categorizing and Predicting Semantic Image-Text Relations

Localizing Adverts in Outdoor Scenes

Contact Info

Product

Resources

About