AttentiveVideo

Abstract

Understanding a target audience's emotional responses to a video advertisement is crucial to evaluate the advertisement's effectiveness. However, traditional methods for collecting such information are slow, expensive, and coarse grained. We propose AttentiveVideo, a scalable intelligent mobile interface with corresponding inference algorithms to monitor and quantify the effects of mobile video advertising in real time. Without requiring additional sensors, AttentiveVideo employs a combination of implicit photoplethysmography (PPG) sensing and facial expression analysis (FEA) to detect the attention, engagement , and sentiment of viewers as they watch video advertisements on unmodified smartphones. In a 24-participant study, AttentiveVideo achieved good accuracy on a wide range of emotional measures (the best average accuracy = 82.6% across nine measures). While feature fusion alone did not improve prediction accuracy with a single model, it significantly improved the accuracy when working together with model fusion. We also found that the PPG sensing channel and the FEA technique have different strength in data availability, latency detection, accuracy, and usage environment. These findings show the potential for both low-cost collection and deep understanding of emotional responses to mobile video advertisements.

RCEA: Real-time, Continuous Emotion Annotation for Collecting Precise Mobile Video Ground Truth Labels

“…This was done given that Circumplex (continuous) and SAM (discrete) are the most widely used, and have shown to exhibit good usability. Fourth, we restricted our work to mobile video trailers from the MAHNOB database [82,97], and do not test other types of content (e.g., MOOC videos [109] or advertisements [78]). While MAHNOB is widely used and contains validated emotion annotation labels, it does limit the CHI 2020, April 25-30, 2020, Honolulu, HI, USA Algorithm 1: Annotation Fusion Input: P∈R I×J for j =1 to J ←number of annotation samples for i = 1 to I ←number of participants Calculate D j of P i j using Eq.…”

“…Our work attempts to tie in together multiple research areas: small mobile form factor, mobility context, capturing emotion experience, and designing for divided attention. One can ask: why not automatically sense behavioral signals (e.g., facial emotional expressions [78]) given that smartphones have front-facing cameras that do not require users to annotate at all? While scientists generally agree that facial movements convey a range of information that serves to express emotional states, to use facial expressions as sole indicators of emotion is misleading.…”

“…short-form [11]. Whether the end goal is to create positive associations with short form video [65], quantify emotion responses to mobile advertisements [78], or improve learning gains in mobile MOOC videos [112], it is important to collect accurate and precise ground truth labels throughout the user's watching experience. However, this poses challenges for real-time and continuous mobile annotation, as performing typical tasks (e.g., mobile video) related to mobile contexts (e.g., crowded bus context) taxes attention and demands users to multi-task [27,75,99], especially since cognitive resources are reserved partly for monitoring the outside context [106].…”

“…This makes them unsuitable for mobile interaction. Facial expressions are also commonly used for continuous emotion annotation [78,95], however in addition to privacy concerns, such expressions do not always overtly show emotion [8]. On the other hand, research on mobile emotion sensing has focused on implicit sensing methods (e.g., touch interactions [69] or typing patterns [36]) to both free the user from manual annotation and sense affective states automatically.…”

Automatic Video Creation From a Web Page

“…There is increasing amount of research on supporting advertisement using computational methods, including modeling audience's visual interests [64] and attention flow [49] or emotion responses to mobile ads [51], and understanding image and video ads [26,62,27]. Beyond commercial ads, VisiBlends introduced techniques to combine objects based on their semantic visuals and constraints in graphical design, which can be used to convey a marketing message [15].…”