Knowledge-infused Learning for Entity Prediction in Driving Scenes

Abstract

Scene understanding is a key technical challenge within the autonomous driving domain. It requires a deep semantic understanding of the entities and relations found within complex physical and social environments that is both accurate and complete. In practice, this can be accomplished by representing entities in a scene and their relations as a knowledge graph (KG). This scene knowledge graph may then be utilized for the task of entity prediction, leading to improved scene understanding. In this paper, we will define and formalize this problem as Knowledge-based Entity Prediction (KEP). KEP aims to improve scene understanding by predicting potentially unrecognized entities by leveraging heterogeneous, high-level semantic knowledge of driving scenes. An innovative neuro-symbolic solution for KEP is presented, based on knowledge-infused learning, which 1) introduces a dataset agnostic ontology to describe driving scenes, 2) uses an expressive, holistic representation of scenes with knowledge graphs, and 3) proposes an effective, non-standard mapping of the KEP problem to the problem of link prediction (LP) using knowledge-graph embeddings (KGE). Using real, complex and high-quality data from urban driving scenes, we demonstrate its effectiveness by showing that the missing entities may be predicted with high precision (0.87 Hits@1) while significantly outperforming the non-semantic/rule-based baselines.

Pedestrian intention estimation and trajectory prediction based on data and knowledge‐driven method

“…Tan et al [32] used a top-down approach to construct a knowledge graph of the urban transportation systems, using the pattern layer of the knowledge graph to achieve knowledge reuse and sharing, completing knowledge complementation based on knowledge inference, and mining the implicit relationships between traffic entities. Wickramarachchi et al [33] argued that the field of autonomous driving was exploring knowledge graphs to manage the large amount of heterogeneous data generated from autonomous vehicles, including data from sensors such as LiDAR. They explored the evaluation data embedded in autonomous driving knowledge graphs and performed related evaluations.…”

Common Sense Knowledge Infusion for Visual Understanding and Reasoning: Approaches, Challenges, and Applications

“…ML models leverage the explicit semantics and factual knowledge in KGs as common sense knowledge, which improves the performance and robustness of the models. 16 The infusion of common sense knowledge using KGs enhances the reasoning capabilities of the models by improving their interpretability. 17 In addition, this also enables the models to alleviate the bias toward generic and frequently occurring concepts and give equal significance to infrequent but important concepts, which improves the recall of the models while maintaining precision.…”

A Survey on Knowledge Graph-based Methods for Automated Driving

“…Wickramarachchi et al [111] generated a KGE from a scene knowledge graph, and use the embedding to predict missing objects in the scene with high accuracy. This is accomplished with a novel mapping and formalization of object detection as a KG link prediction problem.…”