In the automotive industry, researchers, AI experts, and developers are actively pushing deep learning based approaches for In-vehicle safety. In this research paper, we propose a hybrid deep learning based visual system for providing feedback to the driver in a non-intrusive manner. We describe a hybrid SSD-RBM (Single Shot MultiBox Detector - Restricted Boltzmann Machine) model for face feature identification. In this system, object detection, object tracking, and observations are processed through a full pipeline of image processing and detect the driver's movements and generate a safe and efficient action plan in real time. This in-vehicle interactive system assists drivers in regulating driving performance and avoiding hazards.