Deep Learning has shown remarkable success during the last decade for essential tasks in computer vision and natural language processing. Yet, challenges remain in the development and deployment of artificial intelligence (AI) models in real-world cases, such as dependence on extensive data and trust, explainability, traceability, and interactivity. These challenges are amplified in high-risk fields, including healthcare, cyber threats, crisis response, autonomous driving, and future manufacturing. On the other hand, symbolic computing with knowledge graphs has shown significant growth in specific tasks with reliable performance. This tutorial (a) discusses the novel paradigm of knowledge-infused deep learning to synthesize neural computing with symbolic computing (b) describes different forms of knowledge and infusion methods in deep learning, and (c) discusses application-specific evaluation methods to assure explainability and reasoning using benchmark datasets and knowledge-resources. The resulting paradigm of "knowledgeinfused learning" combines knowledge from both domain expertise and physical models. A wide variety of techniques involving shallow, semi-deep, and deep infusion will be discussed along with the corresponding intuitions, limitations, use cases, and applications. More details can be found http://kidl2020.aiisc.ai/.
Scene understanding is a key technical challenge within the autonomous driving domain. It requires a deep semantic understanding of the entities and relations found within complex physical and social environments that is both accurate and complete. In practice, this can be accomplished by representing entities in a scene and their relations as a knowledge graph (KG). This scene knowledge graph may then be utilized for the task of entity prediction, leading to improved scene understanding. In this paper, we will define and formalize this problem as Knowledge-based Entity Prediction (KEP). KEP aims to improve scene understanding by predicting potentially unrecognized entities by leveraging heterogeneous, high-level semantic knowledge of driving scenes. An innovative neuro-symbolic solution for KEP is presented, based on knowledge-infused learning, which 1) introduces a dataset agnostic ontology to describe driving scenes, 2) uses an expressive, holistic representation of scenes with knowledge graphs, and 3) proposes an effective, non-standard mapping of the KEP problem to the problem of link prediction (LP) using knowledge-graph embeddings (KGE). Using real, complex and high-quality data from urban driving scenes, we demonstrate its effectiveness by showing that the missing entities may be predicted with high precision (0.87 Hits@1) while significantly outperforming the non-semantic/rule-based baselines.
Generating high-quality annotations for object detection and recognition is a challenging and important task, especially in relation to safety-critical applications such as autonomous driving (AD). Due to the difficulty of perception in challenging situations such as occlusion, degraded weather, and sensor failure, objects can go unobserved and unlabeled. In this paper, we present CLUE-AD, a general-purpose method for detecting and labeling unobserved entities by leveraging the object continuity assumption within the context of a scene. This method is dataset-agnostic, supporting any existing and future AD datasets. Using a real-world dataset representing complex urban driving scenes, we demonstrate the applicability of CLUE-AD for detecting unobserved entities and augmenting the scene data with new labels.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.