In-cabin occupant monitoring system based on improved Yolo, deep reinforcement learning, and multi-task CNN for autonomous driving

Abstract

Vehicle in-cabin occupant monitoring system is becoming a crucial feature of the automobile industry and challenging research topic to enhance both vehicle safety, security, and comfort of conventional and future intelligent vehicles. Precise information about the number, position, and characteristics of occupants as well as objects located inside the vehicle must be available. Current industrial systems for seat occupancy detection are based on multiple weight sensors, capacitive sensors, electric field, or ultrasonic sensors. They cannot necessarily make the right distinction in borderline cases. A simple pressure sensor cannot tell whether the weight on the seat comes from a person or an inanimate object. Recently, the Artificial Intelligence (AI) based advanced systems have attracted attention for various fields such as automobile industry. Especially, with the advancement of deep learning that has shown very high classification accuracies compared to hand-crafted features on various computer vision tasks. For the above reasons, we propose a new automatic AI occupant monitoring system based on two cameras installed inside the vehicle. Our goal is to develop an automatic detection and recognition system with high accuracy performance, low computational cost and small weight model. Our system fuses our modified deep convolutional network Yolo model and deep reinforcement learning to detect and classify passengers and objects inside the vehicle. It can predict the gender, the age and the emotion of occupants based on our proposed muti task convolutional neural networks. In our end-to-end system, this approach is more efficient time and memory wise by solving all the tasks in the same process and storing a single CNN instead of storing a CNN for each task. Principal applications of our system are intelligent airbag management, seat belt reminder, life presence and in shared cabin preferences. We perform comparative evaluation based on the public datasets SVIRO, TiCaM, Aff-Wild and Adience dataset to demonstrate the superior performance of our proposed system.