Interpretable Global-Local Dynamics for the Prediction of Eye Fixations in Autonomous Driving Scenarios

Abstract

Human eye movements while driving reveal that visual attention largely depends on the context in which it occurs. Furthermore, an autonomous vehicle which performs this function would be more reliable if its outputs were understandable. Capsule Networks have been presented as a great opportunity to explore new horizons in the Computer Vision field, due to their capability to structure and relate latent information. In this paper, we present a hierarchical approach for the prediction of eye fixations in autonomous driving scenarios. Context-driven visual attention can be modeled by considering different conditions which, in turn, are represented as combinations of several spatio-temporal features. With the aim of learning these conditions, we have built an encoder-decoder network which merges visual features' information using a global-local definition of capsules. Two types of capsules are distinguished: representational capsules for features and discriminative capsules for conditions. The latter and the use of eye fixations recorded with wearable eye tracking glasses allow the model to learn both to predict contextual conditions and to estimate visual attention, by means of a multi-task loss function. Experiments show how our approach is able to express either frame-level (global) or pixel-wise (local) relationships between features and contextual conditions, allowing for interpretability while maintaining or improving the performance of black-box related systems in the literature. Indeed, our proposal offers an improvement of 29% in terms of information gain with respect to the best performance reported in the literature. INDEX TERMS Top-down visual attention, eye fixation prediction, context-based learning, interpretability, capsule networks, convolutional neural networks, autonomous driving.

Explainable Intrusion Detection Systems (X-IDS): A Survey of Current Methods, Challenges, and Opportunities

“…To foster the trust of AI systems in the transportation domain, researchers are proposing explanations systems [34]. Some works based on image processing with explainability is found in [35], [36], [37], and [38]. Transparency regarding decision-making processes is critical in the criminal justice system [27], [39].…”

Mining tacit knowledge of pilots in flight missions based on visual cognitive graph

“…In traffic safety domain, a Maximum Entropy Deep Inverse Reinforcement Learning (MEDIRL) framework [36] is used to model the visual attention allocation of drivers in imminent rear-end collisions, which takes information includes traffic density, distance to other cars, and brake lights as context information. Similarly, in [37], Capsule Networks is used to model context-driven visual attention by considering different conditions. However, Artificial Neural Network is difficult to explain, thus is hard to find the hidden knowledge or rules.…”