Use of Machine Learning for Deception Detection From Spectral and Cepstral Features of Speech Signals

Abstract

In this research, four unique nonlinear speech features are extracted and analyzed to study the dissimilarity pattern between when the speaker is being deceitful and truthful based on how human speech is perceived. The speaker was under stress in a police interrogation where two ground truth and two deceitful responses were recorded during three different times of the day. Using the audio recordings from all three sessions, the cepstral features and spectral energy features are extracted. Cepstral features are the Mel frequency cepstrum coefficient, from where the delta cepstrum and the time-difference cepstrum features are developed. On the other hand, the spectral energy features are the energy of Bark band energy from where the delta energy and the time-difference energy features are developed. The Levenberg-Marquardt classification method and the long short-term memory classification method are then applied to evaluate the accuracy of detecting deception based on the nine unique training and testing combinations of the three different sessions and their extracted cepstrum and spectral energy features. In addition, the principal component analysis is applied to reduce the dimensionality from the extracted features for further improvement. The projected principal components of the four types of features showed improved accuracy in order to distinguish between truthful and deceptive speech pattern. After incorporating with principal component analysis, the long short-term memory classification method with time-difference spectral energy feature shows the highest recognition rate compared to Levenberg-Marquardt algorithm with other cepstral and spectral features.

A Comprehensive Review on Features Extraction and Features Matching Techniques for Deception Detection

“…For conduct this comprehensive research, the authors used the database which is a collection of utterances from the audio recording of a male suspect under criminal investigation. The suspect was determined to have given deceptive statements under questioning during polygraph testing [3], [4], [6], [12], [14], [86]. Audio recordings of three sessions of polygraph testing with the same questions by the investigator and the same responses by the suspect will be used for analysis and synthesis.…”

“…Consequently, the articulatory and respiratory movements are affected by speech production [3], [4]. Therefore, it is important to address these issues in a collective way for the experts who works in the fields of law enforcement, education, health care, government agencies, border crossings, military screenings, regular job screenings, telecommunications, informants at embassies and consulates around the world [3]- [6]. It well known that human speech has emotion and nonlinguistic information that encoded in The associate editor coordinating the review of this manuscript and approving it for publication was Donato Impedovo .…”

“…From a psychological perspective, an individual is being deceptive when subconscious or conscious movements present themselves including shortened length of speech, a flushed face, changes in the individual's voice frequency, avoidant eye contact, changes in the diameter of the eye pupil, as well as presenting a more rigid body [4], [5]. Traditional methods have attempted to take advantage of these deceptive indicators to detect deception with a relatively high accuracy rate through the combinatory use of various devices such as the polygraph, cardiovascular activity monitor governed by the sympathetic and parasympathetic nervous systems (i.e., blood pressure), heartbeat sensors, strain gauges to measure respiratory rate, and electrodermal activity (i.e., sweatiness of fingertips) [6]. The polygraph measures some human responses like respiratory rate, electro-dermal activity, heart rate, and others through direct contact, which can lead to numerous challenges and complexities in terms of implementation.…”

“…This paper provides an overview on deception detection techniques and classification methods specifically Linear Regression [1], J48 Decision Tree [1], [72], Multi-Layer Perceptron [1], Gradient Boosting Decision Tree [1], Levenberg-Marquardt [6], [44], BFGS Quasi-Newton [6], Long Short Term Memory [6], [39], [41], [58], Support Vector Machine [1], [13], [15], [39], [41], [45], [46] [58], [79], GMM [13], RVM [15], ensemble [39], Random Forest Decision Tree classifier [73], improved dense trajectories and OpenFace [79] and Ripper rule induction classifier [74]. In addition, non-verbal features like thermal facial analysis [7], [23], [32], [36], [38] eye movement analysis [8], [22], [26]- [28], visual cues [9], [10], text analysis with BERT [16], brain activity analysis [21], [33], [35], multimodal features [24], [29], infrared imaging [31], and facial expressions…”

“…In addition, non-verbal features like thermal facial analysis [7], [23], [32], [36], [38] eye movement analysis [8], [22], [26]- [28], visual cues [9], [10], text analysis with BERT [16], brain activity analysis [21], [33], [35], multimodal features [24], [29], infrared imaging [31], and facial expressions [18], [25], [27], [30], [37] have been used for deception detection research and reported encouraging results. Similarly, verbal speech features such as acoustic [11], STE [1], [46], time-difference energy [6], delta energy [6], energy [39], [40], MFCC and energy [39], zero crossing rate [40], [45], [46], entropy of energy [40], spectral centroid [40], spectral spread [40], spectral entropy [40], spectral flux [40], [75], spectral roll-off [40], chroma vector [40], chroma deviation [40], significant energy [42], [44...…”