Is neuro-symbolic AI meeting its promises in natural language processing? A structured review

Abstract

Advocates for Neuro-Symbolic Artificial Intelligence (NeSy) assert that combining deep learning with symbolic reasoning will lead to stronger AI than either paradigm on its own. As successful as deep learning has been, it is generally accepted that even our best deep learning systems are not very good at abstract reasoning. And since reasoning is inextricably linked to language, it makes intuitive sense that Natural Language Processing (NLP), would be a particularly well-suited candidate for NeSy. We conduct a structured review of studies implementing NeSy for NLP, with the aim of answering the question of whether NeSy is indeed meeting its promises: reasoning, out-of-distribution generalization, interpretability, learning and reasoning from small data, and transferability to new domains. We examine the impact of knowledge representation, such as rules and semantic networks, language structure and relational structure, and whether implicit or explicit reasoning contributes to higher promise scores. We find that systems where logic is compiled into the neural network lead to the most NeSy goals being satisfied, while other factors such as knowledge representation, or type of neural architecture do not exhibit a clear correlation with goals being met. We find many discrepancies in how reasoning is defined, specifically in relation to human level reasoning, which impact decisions about model architectures and drive conclusions which are not always consistent across studies. Hence we advocate for a more methodical approach to the application of theories of human reasoning as well as the development of appropriate benchmarks, which we hope can lead to a better understanding of progress in the field. We make our data and code available on github for further analysis.11 https://github.com/kyleiwaniec/neuro-symbolic-ai-systematic-review

A Survey on Verification and Validation, Testing and Evaluations of Neurosymbolic Artificial Intelligence

“…There are multiple recent surveys that cover neurosymbolic AI and its applications in general [17]- [24] and surveys that focus on more specific applications such as graph structures [25], biomedical knowledge graphs [26], or natural language processing [27]. None of the just mentioned surveys covers testing, validation or verification in the domain of neurosymbolic AI and we could not find any surveys covering this topic to this date.…”

Enhancing Contextual Understanding of Mistral LLM with External Knowledge Bases

“…The field of knowledge representation and reasoning provides critical insights into how information can be structured and utilized by AI systems to simulate human-like understanding and decision-making processes [47,48,14,21]. Research in this area explores various frameworks and methodologies for organizing knowledge in formats that are accessible and interpretable by computational models [17,49,48,50].…”

“…Despite advancements in creating more sophisticated representation schemas, there remains a gap in seamlessly integrating these structured knowledge forms with the operational mechanisms of AI models, which highlights the complexity of enabling AI to leverage external knowledge for enhanced reasoning capabilities, suggesting an area ripe for further investigation and development. Recent advancements in graph-based knowledge representation have facilitated the development of more dynamic and interconnected knowledge structures, enabling AI systems to perform complex reasoning with greater efficiency [47,53,54]. The integration of semantic web technologies has also shown potential in enhancing the interoperability between AI models and external knowledge bases, promoting a more unified approach to knowledge utilization [43,55,56].…”

“…Addressing these challenges is essential for fully realizing the benefits of external knowledge integration in enhancing AI's contextual understanding and factual grounding [17,61,68]. Recent explorations into hybrid models combining neural networks with external knowledge bases have shown promise in bridging the gap between AI's internal reasoning capabilities and the external world's complexity [47,69,26]. Studies employing transformer-based architectures to dynamically query and incorporate information from external sources have reported notable enhancements in the model's ability to generate contextually rich and accurate responses [70,71].…”

Integrating Deep Learning with Symbolic Reasoning in TinyLlama for Accurate Information Retrieval

“…Symbolic reasoning in AI has been explored as a means to incorporate logic and rule-based decision-making into intelligent systems. Research has been conducted on the integration of symbolic reasoning to improve the interpretability of AI models, enabling a better understanding of how models arrive at certain conclusions [10], [31]. The development of hybrid systems that combine machine learning with symbolic reasoning has been a key focus, aiming to leverage the strengths of both approaches [31], [32], [33].…”