Drowsiness Detection Using Facial Expression Features

Abstract

This paper presents the method of detecting driver's drowsiness level from the facial expression. The motivation for this research is to realize the novel safety system which can detect the driver's slight drowsiness and keep the driver awake while driving. The brain wave is commonly used as the drowsiness index. However, it is not suitable for the in-vehicle system since it is measured with sensors worn over the head. We precisely investigated the relationship between the change of brain wave and other drowsiness indices that can be measured without any contact; PERCLOS, heart rate, lane deviation, and facial expression. We found that the facial expression index had the highest linear correlation with the brain wave. Therefore, we selected the facial expression as the drowsiness-detection index and automated the drowsiness detection from the facial expression. Three problems need to be solved for automation; (1) how to de ne the features of drowsy expression, (2) how to capture the features from the driver's video-recorded facial image, and (3) how to estimate the driver's drowsiness index from the features. First, we found that frontalis muscle, zygomaticus major muscle, and masseter muscle activated with increase of drowsiness in more than 75 percents of participants. According to the result, we determined the coordinates data of points on eyebrows, eyelids, and mouth as the features of drowsiness expression. Second, we calculated the 3D coordinates data of the features by image processing with Active Appearance Model (AAM). Third, we applied k-Nearest-Neighbor method to classify the driver's drowsiness level. Eleven participants' data of the features and the drowsiness level estimated by trained observers were used as the training data. We achieved the classi cation of the drivers' drowsiness in a driving simulator into 6 levels. The average Root Mean Square Errors (RMSE) among 12 participants was less than 1.0 level.

Comprehensive Drowsiness Level Detection Model Combining Multimodal Information

“…2 shows the correspondence between the KSS values and the self-reported drowsiness levels, indicating the correspondence frequencies before and after driving by blue circles and red circles, respectively. The high correspondence between KSS 7 and drowsiness level 3 is consistent with previous results on driving performance that show steep degradation starting at KSS 7 [34], [35] and drowsiness level 3 [11], [12].…”

“…The average F1 score for the drowsiness levels was designated as the estimation accuracy of the entire model. In addition, to compare with the existing method [11] which uses RMSE as the evaluation metric, the estimated results for the drowsiness rating values of each subject were also evaluated using RMSE as the accuracy evaluation of the entire model.…”

“…Table I shows the corresponding features of these levels. The drowsiness level is also widely used, because the level determined by trained observers based on the driver's facial expressions shows a high correlation with the driver's self-reported drowsiness level and can thus be used as a substitute for subjective reports [11], [12].…”

Detection of Driver's Drowsy Facial Expression

“…1 illustrates the scenes recorded during the experiment. Kitajima's facial expression estimation method [1] is often used to estimate a driver's drowsiness degree [9,15,16]. In this method, the drowsiness degree was divided into five levels on the basis of the captured video: "level 1: not sleepy," "level 2: slightly sleepy," "level 3: sleepy," "level 4: rather sleepy," "level 5: very sleepy."…”

“…Finally, after analyzing a video of captured facial expressions, Hachisuka et al [15,16] focused on the movement of facial feature points on the eyes, eyebrows, and mouth. In these studies, the drowsiness degree was evaluated on the basis of changes in these points with reference to an awakened state.…”