Multimodal Knowledge Graph for Deep Learning Papers and Code

Abstract

Keeping up with the rapid growth of Deep Learning (DL) research is a daunting task. While existing scientific literature search systems provide text search capabilities and can identify similar papers, gaining an in-depth understanding of a new approach or an application is much more complicated. Many publications leverage multiple modalities to convey their findings and spread their ideasthey include pseudocode, tables, images and diagrams in addition to text, and often make publicly accessible their implementations. It is important to be able to represent and query them as well. We utilize RDF Knowledge graphs (KGs) to represent multimodal information and enable expressive querying over modalities. In our demo we present an approach for extracting KGs from different modalities, namely text, architecture images and source code. We show how graph queries can be used to get insights into different facets (modalities) of a paper, and its associated code implementation. Our innovation lies in the multimodal nature of the KG we create. While our work is of direct interest to DL researchers and practitioners, our approaches can also be leveraged in other scientific domains.

Analysing the requirements for an Open Research Knowledge Graph: use cases, quality requirements, and construction strategies

“…It contains five entity types (tasks, methods, metrics, materials, others) linked by 27 relations types. Kannan et al [58] create a multimodal KG for deep learning papers from text and images and the corresponding source code. Brack et al [17] generate a KG for 10 different science domains with the concept types material, method, process, and data.…”

Research on the Construction and Application of Breast Cancer-Specific Database System Based on Full Data Lifecycle

“…Meanwhile, the entity is not limited to the single representation only in breast cancer-related terms. The traditional knowledge graph is out of use, and the current knowledge graph integrates multi-modal knowledge, and displays, represents and utilizes medical data in various forms to the largest extent for the convenience of learning and understanding by researchers ( 17 , 18 ). The partial knowledge graph constructed is shown in Figure 6 .…”

Essential Features in a Theory of Context for Enabling Artificial General Intelligence

“…COVID-19 knowledge graphs were even constructed in the earliest days of the pandemic [67]. Typically, KGs are constructed over text and Web data, though more recently, multi-modal and domain-specific KG construction has started to become popular [68,69]. KG research has a closer connection to context-rich AI than may initially meet the eye.…”