Learning to Steer by Mimicking Features from Heterogeneous Auxiliary Networks

Abstract

The training of many existing end-to-end steering angle prediction models heavily relies on steering angles as the supervisory signal. Without learning from much richer contexts, these methods are susceptible to the presence of sharp road curves, challenging traffic conditions, strong shadows, and severe lighting changes. In this paper, we considerably improve the accuracy and robustness of predictions through heterogeneous auxiliary networks feature mimicking, a new and effective training method that provides us with much richer contextual signals apart from steering direction. Specifically, we train our steering angle predictive model by distilling multi-layer knowledge from multiple heterogeneous auxiliary networks that perform related but different tasks, e.g., image segmentation or optical flow estimation. As opposed to multitask learning, our method does not require expensive annotations of related tasks on the target set. This is made possible by applying contemporary off-the-shelf networks on the target set and mimicking their features in different layers after transformation. The auxiliary networks are discarded after training without affecting the runtime efficiency of our model. Our approach achieves a new state-of-the-art on Udacity and Comma.ai, outperforming the previous best by a large margin of 12.8% and 52.1% 1 , respectively. Encouraging results are also shown on Berkeley Deep Drive (BDD) dataset.

Multi‐task deep learning with optical flow features for self‐driving cars

“…Building around the optical flow feature, we propose a deep learning system to generate the control signals. Because the flow can encode the temporal information between two frames, our system does not need a recurrent module [9,10,12], although we can include such modules for considering a longer temporal duration. The frame-based nature of our system allows the system to react quickly to unexpected events.…”

“…They separated the flow into the image velocity component and the object motion using the camera motion estimated from a deep network [31]. Hou et al [10] proposed to utilize two pre-trained auxiliary networks, one for image segmentation and one for optical flow prediction, to guide the main encoder network to generate low-dimensional deep feature, which is then inputted to an LSTM module for predicting the control signals. We believe that the use of the LSTM module to model temporal information may not be necessary, as the optical flow features already define the movement feature across time effectively.…”

“…We therefore demonstrate that by using optical flow directly as the input, control signals can be accurately predicted. We further propose a multi-task structure that utilizes two sub-branches to predict two different control signals, the steering angle and speed, as opposed to [10] that outputs them in the same layer. Due to the use of individual subbranches, our system has a better capacity to generalize to control signals in which angle and speed are in some unusual combinations.…”

“…More recent works leverage auxiliary tasks for feature learning. Tasks such as segmentation [9] and optical flow estimation [10] can be applied to enhance the quality of training for the vehicle control prediction problem. Furthermore, multi-task frameworks have been shown to be effective in controlling multiple aspects of driving at the same time.…”

“…In particular, temporal information has been taking into consideration for further improving the robustness in autonomous driving. Some researchers proposed using Long Short-Term Memory (LSTM) in their deep learning frameworks to infer steering angles [9,10,12]. However, while these methods work well under normal circumstances, relying on a period of temporal information for driving controls could be risky as unexpected discrete events such as a car crash can happen instantaneously.…”