Real-time Emotion Recognition for Sales

Abstract

Positive emotion is a pre-condition to any sales contract. Likewise, the ability to perceive the emotions of a customer impacts sales performance.To support emotional perception in buyer-seller interactions, we propose an audio-visual emotion recognition system that can recognize eight emotions: neutral, calm, sad, happy, angry, fearful, surprised, and disgusted. We reduced noise in audio samples and we applied transfer learning for image feature extraction based on a pre-trained deep neural network VGG16. For emotion recognition, we successfully obtained an audio emotion-recognition accuracy of 62.51% and 68% and video emotion-recognition accuracy of 97.13% and 97.77% on the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) and Surrey Audio-Visual Expressed Emotion (SAVEE) datasets respectively. For the combination of the two models, our proposed merging mechanism without re-training achieved an accuracy of close to 100% on both datasets. Finally, we demonstrated our system for a customer satisfaction use case in a real customer-to-salesperson interaction using audio and video models, achieving an average accuracy of 78%.

Facial expression recognition based on adaptation of the classifier to videos of the user

“…Понимание эмоционального состояния людей может быть важным во многих сферах, например, для предотвращения возможных конфликтных ситуаций по данным видеонаблюдения [5], определения внимания и стресса водителя во время движения [6], отслеживания изменений в эмоциях человека в зависимости от окружения и различных внешних событий [7], определения реакции покупателей на рекламную компанию [8], в рекомендательных системах [9] и т.п.…”