Fusion of heart rate variability and pulse rate variability for emotion recognition using lagged poincare plots

Abstract

Designing an efficient automatic emotion recognition system based on physiological signals has attracted great interests within the research of human-machine interactions. This study was aimed to classify emotional responses by means of a simple dynamic signal processing technique and fusion frameworks. The electrocardiogram and finger pulse activity of 35 participants were recorded during rest condition and when subjects were listening to music intended to stimulate certain emotions. Four emotion categories, including happiness, sadness, peacefulness, and fear were chosen. Estimating heart rate variability (HRV) and pulse rate variability (PRV), 4 Poincare indices in 10 lags were extracted. The support vector machine classifier was used for emotion classification. Both feature level (FL) and decision level (DL) fusion schemes were examined. Significant differences have been observed between lag 1 Poincare plot indices and the other lagged measures. The mean accuracies of 84.1, 82.9, 79.68, and 76.05% were obtained for PRV, DL, FL, and HRV measures, respectively. However, DL outperformed others in discriminating sadness and peacefulness, using SD and total features, correspondingly. In both cases, the classification rates improved up to 92% (with the sensitivity of 95% and specificity of 83.33%). Totally, DL resulted in better performances compared to FL. In addition, the impact of the fusion rules on the classification performances has been confirmed.

Different Types of Sounds and Their Relationship With the Electrocardiographic Signals and the Cardiovascular System – Review

“…It is possible to see that most selected studies used a sample with healthy subjects ( Holand et al, 1999 ; Martin-Soelch et al, 2006 ; Uclés et al, 2006 ; Koelsch et al, 2007 ; Kasprzak, 2010 ; Lee et al, 2010 ; Dousty et al, 2011 ; Sancini et al, 2012 ; Chen et al, 2014b ; Naji et al, 2014b ; Pérez-Lloret et al, 2014 ; da Silva and Backs, 2015 ; Krabs et al, 2015 ; Nardelli et al, 2015 ; Nozaki et al, 2015 ; Sim et al, 2015 ; Watanabe et al, 2015 ; Nakajima et al, 2016 ; El Aarbaoui et al, 2017 ; Goshvarpour et al, 2017 ; Ishimitsu et al, 2017 ). Samples are also used with a combination of males and females ( Martin-Soelch et al, 2006 ; Björ et al, 2007 ; Koelsch et al, 2007 ; Lee et al, 2010 ; Orini et al, 2010 ; Kraus et al, 2013 ; Naji et al, 2014b ; da Silva and Backs, 2015 ; Nozaki et al, 2015 ; Bidin et al, 2016 ; El Aarbaoui et al, 2017 ; Goshvarpour et al, 2017 ). There were few studies with just males ( Tomei et al, 2000 ; Cho et al, 2011 ; Gupta and Gupta, 2015 ; Walker et al, 2016 ) or just females ( Binns-Turner et al, 2011 ; Rhomberg et al, 2014 ; Nakajima et al, 2016 ; Abedi et al, 2017 ).…”

“…There were few studies with just males ( Tomei et al, 2000 ; Cho et al, 2011 ; Gupta and Gupta, 2015 ; Walker et al, 2016 ) or just females ( Binns-Turner et al, 2011 ; Rhomberg et al, 2014 ; Nakajima et al, 2016 ; Abedi et al, 2017 ). With respect to the sample size, many of the studies used a size between 20 and 33 ( Holand et al, 1999 ; Thayer and Faith, 2001 ; Chang et al, 2004 ; Martin-Soelch et al, 2006 ; Björ et al, 2007 ; Kasprzak, 2010 ; Krantz et al, 2010 ; Binns-Turner et al, 2011 ; Cho et al, 2011 ; Dousty et al, 2011 ; Croy et al, 2013 ; Naji et al, 2014b ; Pérez-Lloret et al, 2014 ; Nardelli et al, 2015 ; Nozaki et al, 2015 ; Al-Galal et al, 2016 ; Abedi et al, 2017 ) or 35 and 88 ( Tomei et al, 2000 ; Kenntner-Mabiala et al, 2007 ; Koelsch et al, 2007 ; Raggam et al, 2007 ; Graham et al, 2009 ; Orini et al, 2010 ; Sancini et al, 2012 ; Rhomberg et al, 2014 ; Gruhlke et al, 2015 ; Gupta and Gupta, 2015 ; Hajizadeh et al, 2015 ; Krabs et al, 2015 ; Sim et al, 2015 ; Watanabe et al, 2015 ; Gallasch et al, 2016 ; Goshvarpour et al, 2016a , 2017 ; El Aarbaoui et al, 2017 ) subjects. There were few studies with a sample larger than 100 subjects ( Thayer and Faith, 2001 ; Yuan et al, 2005 ; Uclés et al, 2006 ; Goyal et al, 2010 ; Kraus et al, 2013 ; Osiris et al, 2014 ; Tan et al, 2015 ).…”

“…There were few studies with a sample larger than 100 subjects ( Thayer and Faith, 2001 ; Yuan et al, 2005 ; Uclés et al, 2006 ; Goyal et al, 2010 ; Kraus et al, 2013 ; Osiris et al, 2014 ; Tan et al, 2015 ). Similarly, most of the research considered subjects between 18 and 41 ( Yuan et al, 2005 ; Björ et al, 2007 ; Kenntner-Mabiala et al, 2007 ; Goyal et al, 2010 ; Kasprzak, 2010 ; Lee et al, 2010 ; Orini et al, 2010 ; Cho et al, 2011 ; Croy et al, 2013 ; Rhomberg et al, 2014 ; da Silva and Backs, 2015 ; Nardelli et al, 2015 ; Oh et al, 2015 ; Watanabe et al, 2015 ; Goshvarpour et al, 2016a , 2017 ; Ishimitsu et al, 2017 ), and a few studies used subjects older than 42 ( Holand et al, 1999 ; Gupta and Gupta, 2015 ) as part of the sample.…”

“…It is interesting to note that some studies asked subjects to close their eyes ( Naji et al, 2014b ; Nardelli et al, 2015 ; Nozaki et al, 2015 ; Al-Galal et al, 2016 ; Goshvarpour et al, 2016a , b ) to concentrate on presented stimulus. A seated position was also used ( Martin-Soelch et al, 2006 ; Björ et al, 2007 ; Lee et al, 2010 ; Cho et al, 2011 ; Naji et al, 2014b ; Pérez-Lloret et al, 2014 ; Nardelli et al, 2015 ; Sim et al, 2015 ; Watanabe et al, 2015 ; Al-Galal et al, 2016 ; Nakajima et al, 2016 ; Ishimitsu et al, 2017 ) more frequently with respect to a supine position ( Holand et al, 1999 ; Hajizadeh et al, 2015 ; Goshvarpour et al, 2016a , 2017 ; Abedi et al, 2017 ). Moreover, in some cases, some environmental elements were controlled ( Björ et al, 2007 ; Sancini et al, 2012 ; Naji et al, 2014b ; Nardelli et al, 2015 ; Nozaki et al, 2015 ; Sim et al, 2015 ; Watanabe et al, 2015 ; Al-Galal et al, 2016 ; Goshvarpour et al, 2016b ).…”

Survey on AI-Based Multimodal Methods for Emotion Detection

“…Accuracy of the emotion recognition and classification based on physiological signals identification have improved significantly. Research by Goshvarpour et al using HRV and PRV, and fusion rules (feature level, decision level) showed classification rates improved up to 92% (sensitivity: 95%, specificity: 83.33%) [63]. Previous studies have provided evidence for general recognition rate from 57,77% to 85,46% for different emotional states, that more higher set of analyzed signals and more recognized emotions provides better results [64][65][66].…”