Reading Food Experiences from the Face: Effects of Familiarity and Branding of Soy Sauce on Facial Expressions and Video-Based RPPG Heart Rate

Abstract

Food experiences are not only driven by the food’s intrinsic properties, such as its taste, texture, and aroma, but also by extrinsic properties such as visual brand information and the consumers’ previous experiences with the foods. Recent developments in automated facial expression analysis and heart rate detection based on skin color changes (remote photoplethysmography or RPPG) allow for the monitoring of food experiences based on video images of the face. RPPG offers the possibility of large-scale non-laboratory and web-based testing of food products. In this study, results from the video-based analysis were compared to the more conventional tests (scores of valence and arousal using Emojis and photoplethysmography heart rate (PPG)). Forty participants with varying degrees of familiarity with soy sauce were presented with samples of rice and three commercial soy sauces with and without brand information. The results showed that (1) liking and arousal were affected primarily by the specific tastes, but not by branding and familiarity. In contrast, facial expressions were affected by branding and familiarity, and to a lesser degree by specific tastes. (2) RPPG heart rate and PPG both showed effects of branding and familiarity. However, RPPG heart rate needs further development because it underestimated the heart rate compared to PPG and was less sensitive to changes over time and with activity (viewing of brand information and tasting). In conclusion, this study suggests that recording of facial expressions and heart rates may no longer be limited to laboratories but can be done remotely using video images, which offers opportunities for large-scale testing in consumer science.

Predicting Perceived Hedonic Ratings through Facial Expressions of Different Drinks

“…The intensity of these emotions ranges from 0 to 1. Although this software is not fully accurate in its emotion recognition performance [ 6 , 18 ], the analyses of these emotions have been effectively employed in various experimental situations in food research [ 5 , 19 , 20 , 21 , 22 , 27 , 32 ]. In our previous study [ 30 ], we used a different facial expression analysis software.…”

“…Implicit evaluation methods are diverse [ 4 ], encompassing heart rate variability [ 5 , 6 , 7 ], skin temperature [ 8 ], skin blood flow [ 9 , 10 ], skin conductance response [ 11 ], facial expressions [ 3 , 6 , 12 , 13 ], and brain activity [ 14 , 15 , 16 , 17 ]. Among these, facial expression analysis stands out as a relatively straightforward and practical approach, utilizing readily available automatic classifiers for facial affect recognition [ 18 ].…”

“…Numerous studies have examined the facial expressions of individuals while consuming various foods and beverages, extending beyond basic taste solutions [ 5 , 6 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 ]. Recent research has placed emphasis on predictive modeling based on these analyses, encompassing the prediction of consumer preferences [ 27 ], consumer acceptance of food products [ 28 , 29 ], and even choices in beer selection [ 25 ].…”

What is the link between personality and food behavior?

“…Finally, here, it is worth noting how there may also be a link here between the stereotypical facial expressions that are associated with experiencing the different basic tastes (see Steiner, 1974 , 1979 ; Steiner et al, 2001 ; Spence, 2012 ; Weiland et al, 2010 ) and those that are associated with displays of facial emotion (e.g., Liang et al, 2021 ; cf. de Wijk et al, 2021 ; Greimel et al, 2006 ). 20 This connection certainly makes sense in terms of the increasingly-popular theories of embodied cognition (see Wilson, 2002 ).…”

Novel Techniques to Measure the Sensory, Emotional, and Physiological Responses of Consumers toward Foods

“…Consumers' purchasing decisions are not only affected by the intrinsic sensory properties of the products but also by the extrinsic information that is provided at the time of purchasing. De Wijk et al [5] evaluated the effects of familiarity and branding on the sensory experiences of soy sauces using facial expressions and video-based heart rate measurements. They found that liking and arousal toward the products were affected by taste but not by branding and familiarity.…”