Integration of numeric and symbolic information for semantic image interpretation

Donadello, Ivan; Serafini, Luciano

doi:10.3233/ia-160093

Cited by 11 publications

(9 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…a) Symbolic Models: As complex reasoning requires discrete reasoning operations over reasoning units, most previous studies for complex reasoning are symbolic expert systems [50,51,52,53]. They design a set of rules or templates to identify basic units for reasoning as symbols, such as the quantities and arithmetic signs for numerical reasoning [54,55,56], relevant triples and paths from knowledge graph for commonsense reasoning [57], etc. They then perform deterministic and explicit inferences upon discrete elementary units to predict the answer.…”

Section: B Advanced Methods Of Complex Reasoningmentioning

confidence: 99%

From LSAT: The Progress and Challenges of Complex Reasoning

Wang¹,

Liu²,

Zhong³

et al. 2021

Preprint

View full text Add to dashboard Cite

Complex reasoning aims to draw a correct inference based on complex rules. As a hallmark of human intelligence, it involves a degree of explicit reading comprehension, interpretation of logical knowledge and complex rule application. In this paper, we take a step forward in complex reasoning by systematically studying the three challenging and domain-general tasks of the Law School Admission Test (LSAT), including analytical reasoning, logical reasoning and reading comprehension. We propose a hybrid reasoning system to integrate these three tasks and achieve impressive overall performance on the LSAT tests. The experimental results demonstrate that our system endows itself a certain complex reasoning ability, especially the fundamental reading comprehension and challenging logical reasoning capacities. Further analysis also shows the effectiveness of combining the pre-trained models with the task-specific reasoning module, and integrating symbolic knowledge into discrete interpretable reasoning steps in complex reasoning. We further shed a light on the potential future directions, like unsupervised symbolic knowledge extraction, model interpretability, few-shot learning and comprehensive benchmark for complex reasoning.

show abstract

Section: B Advanced Methods Of Complex Reasoningmentioning

confidence: 99%

From LSAT: The Progress and Challenges of Complex Reasoning

Wang¹,

Liu²,

Zhong³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Nonetheless, other researchers have built upon MS COCO in order to create scene graph datasets, such as those which will be discussed next. • Semantic PASCAL-Part [12] is an OWL conversion of an earlier PASCAL-Part dataset [8]. Formal ontological constructs are used to define each object class, which makes this dataset appropriate for object classification tasks based on detection of constituent parts.…”

Section: Scene Graph Datasetsmentioning

confidence: 99%

OG-SGG: Ontology-Guided Scene Graph Generation. A Case Study in Transfer Learning for Telepresence Robotics

Amodeo¹,

Caballero²,

Díaz-Rodríguez³

et al. 2022

Preprint

View full text Add to dashboard Cite

Scene graph generation from images is a task of great interest to applications such as robotics, because graphs are the main way to represent knowledge about the world and regulate human-robot interactions in tasks such as Visual Question Answering (VQA). Unfortunately, its corresponding area of machine learning is still relatively in its infancy, and the solutions currently offered do not specialize well in concrete usage scenarios. Specifically, they do not take existing "expert" knowledge about the domain world into account; and that might indeed be necessary in order to provide the level of reliability demanded by the use case scenarios. In this paper, we propose an initial approximation to a framework called Ontology-Guided Scene Graph Generation (OG-SGG), that can improve the performance of an existing machine learning based scene graph generator using prior knowledge supplied in the form of an ontology; and we present results evaluated on a specific scenario founded in telepresence robotics.

show abstract

“…In this work, we use a curated version of the PASCAL-Part provided by [28] 10 . The idea is to reduce the number of elements of the original PASCAL-Part by collapsing categories together such as "upper arm" and "lower arm" inside a single category "arm".…”

Section: Appendixmentioning

confidence: 99%

EXplainable Neural-Symbolic Learning (X-NeSyL) methodology to fuse deep learning representations with expert knowledge graphs: the MonuMAI cultural heritage use case

Díaz-Rodríguez,

Lamas,

Sanchez

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The latest Deep Learning (DL) models for detection and classification have achieved an unprecedented performance over classical machine learning algorithms. However, DL models are black-box methods hard to debug, interpret, and certify. DL alone cannot provide explanations that can be validated by a non technical audience such as end-users or domain experts. In contrast, symbolic AI systems that convert concepts into rules or symbols -such as knowledge graphs-are easier to explain. However, they present lower generalisation and scaling capabilities. A very important challenge is to fuse DL representations with expert knowledge. One way to address this challenge, as well as the performance-explainability trade-off is by leveraging the best of both streams without obviating domain expert knowledge. In this paper, we tackle such problem by considering the symbolic knowledge is expressed in form of a domain expert knowledge graph. We present the eXplainable Neural-symbolic learning (X-NeSyL) methodology, designed to learn both symbolic and deep representations, together with an explainability metric to assess the level of alignment of machine and human expert explanations. The ultimate objective is to fuse DL representations with expert domain knowledge during the learning process so it serves as a sound basis for explainability. In particular, X-NeSyL methodology involves the concrete use of two notions of explanation, both at inference and training time respectively: 1) EXPLANet: Expert-aligned eXplainable Part-based cLAssifier NETwork Architecture, a compositional convolutional neural network that makes use of symbolic representations, and 2) SHAP-Backprop, an explainable AI-informed training procedure that corrects and guides the DL process to align with such symbolic representations in form of knowledge graphs. We showcase X-NeSyL methodology using MonuMAI dataset for monument facade image classification, and demonstrate that our approach improves not only explainability of DL models but also performance.

show abstract

Integration of numeric and symbolic information for semantic image interpretation

Cited by 11 publications

References 26 publications

From LSAT: The Progress and Challenges of Complex Reasoning

From LSAT: The Progress and Challenges of Complex Reasoning

OG-SGG: Ontology-Guided Scene Graph Generation. A Case Study in Transfer Learning for Telepresence Robotics

EXplainable Neural-Symbolic Learning (X-NeSyL) methodology to fuse deep learning representations with expert knowledge graphs: the MonuMAI cultural heritage use case

Contact Info

Product

Resources

About