Judging the emotional states of customer service staff in the workplace: A multimodal dataset analysis

Abstract

BackgroundEmotions play a decisive and central role in the workplace, especially in the service-oriented enterprises. Due to the highly participatory and interactive nature of the service process, employees’ emotions are usually highly volatile during the service delivery process, which can have a negative impact on business performance. Therefore, it is important to effectively judge the emotional states of customer service staff.MethodsWe collected data on real-life work situations of call center employees in a large company. Three consecutive studies were conducted: first, the emotional states of 29 customer service staff were videotaped by wide-angle cameras. In Study 1, we constructed scoring criteria and auxiliary tools of picture-type scales through a free association test. In Study 2, two groups of experts were invited to evaluate the emotional states of customer service staff. In Study 3, based on the results in Study 2 and a multimodal emotional recognition method, a multimodal dataset was constructed to explore how each modality conveys the emotions of customer service staff in workplace.ResultsThrough the scoring by 2 groups of experts and 1 group of volunteers, we first developed a set of scoring criteria and picture-type scales with the combination of SAM scale for judging the emotional state of customer service staff. Then we constructed 99 (out of 297) sets of stable multimodal emotion datasets. Based on the comparison among the datasets, we found that voice conveys emotional valence in the workplace more significantly, and that facial expressions have more prominant connection with emotional arousal.ConclusionTheoretically, this study enriches the way in which emotion data is collected and can provide a basis for the subsequent development of multimodal emotional datasets. Practically, it can provide guidance for the effective judgment of employee emotions in the workplace.

The Chinese customers and service staff interactive affective system (CCSIAS): introduction to a multimodal stimulus dataset

“…The emotional interaction between customers and service staff refers to a process of emotional convergence, that is, the tendency of one person to imitate the facial expressions, body actions, and communication voices of the other when influenced by the other's emotional state (Hatfield et al, 1993;Belkin, 2009;Zhang et al, 2016;Fang et al, 2019;Liu et al, 2022). Emotional interactions are central to service quality and performance, and play an invaluable role in service delivery (Liao et al, 2009;Liu et al, 2019).…”

“…Emotional interactions are central to service quality and performance, and play an invaluable role in service delivery (Liao et al, 2009;Liu et al, 2019). As such, studies of emotional interactions have received considerable scholars' attention in recent years (Rueff-Lopes et al, 2015;Grandey and Melloy, 2017;Liu et al, 2022;VanKleef and Cote, 2022). Primitive emotional contagion (see Figure 1) means that people pick up verbal and non-verbal cues from others through the sensory system (Hatfield et al, 1993), and then automatically and unconsciously imitate the emotional states of others (Du et al, 2011), thus achieving emotional assimilation (Hennig-Thurau et al, 2006;Zhang et al, 2016).…”

“…Primitive emotional contagion (see Figure 1) means that people pick up verbal and non-verbal cues from others through the sensory system (Hatfield et al, 1993), and then automatically and unconsciously imitate the emotional states of others (Du et al, 2011), thus achieving emotional assimilation (Hennig-Thurau et al, 2006;Zhang et al, 2016). With the high-interaction and high-participation nature of service delivery (Rueff-Lopes et al, 2015;Liu et al, 2022), service-staff are often asked to manage and express their emotions in line with organizational goals. This phenomenon has been characterized as emotional labor (Hochschild, 1979;Ni and Li, 2021), which requires the engagement of higher cognitive systems such as the brain (Zhang and Lu, 2013).…”

“…In terms of data storage structure, if a piece of emotional information can be encoded and stored in a single methodology and can be interpreted by a single sensory system, then this emotional data can be identified as single-modal. However, if the emotional information is encoded and stored in two or more methodologies and must be interpreted by more than one sensory system, the emotional data is multimodal (Kress, 2010;Lahat et al, 2015;Wang et al, 2021;Liu et al, 2022). To date, a series of studies in organizational behavior, marketing and tourism research have confirmed that reliance on single-modal data…”

Multimodality in language education: implications of a multimodal affective perspective in foreign language teaching

“…Emotions are complex and expressed through various modalities, including facial expressions, gestures, and oral expressions ( Liu P. et al, 2022 ). Scholars agree that emotions consist of multiple components and are thus complex to study ( Gao and Cui, 2022 ; Liu et al, 2023 ; Wen et al, 2023 ).…”