A Discussion on the Design of Graph Database Benchmarks

Abstract

Graph Database Management systems (GDBs) are gaining popularity. They are used to analyze huge graph datasets that are naturally appearing in many application areas to model interrelated data. The objective of this paper is to raise a new topic of discussion in the benchmarking community and allow practitioners having a set of basic guidelines for GDB benchmarking. We strongly believe that GDBs will become an important player in the market field of data analysis, and with that, their performance and capabilities will also become important. For this reason, we discuss those aspects that are important from our perspective, i.e. the characteristics of the graphs to be included in the benchmark, the characteristics of the queries that are important in graph analysis applications and the evaluation workbench.

The shortest path algorithm performance comparison in graph and relational database on a transportation network

“…The graph theory uses nodes for storing entities and edges for relationships among them. Graph databases emphasize the relations among entities rather than entities themselves [18]. Any model can be thought of as a representation of reality.…”

“…To our knowledge there are only a handful of graph database benchmarks and guidelines [9,18,24]. Dominguez-Sal et al [18] propose guidelines for a graph oriented benchmark differing from any standard or spatial database benchmark.…”

“…Dominguez-Sal et al [18] propose guidelines for a graph oriented benchmark differing from any standard or spatial database benchmark. According to these authors the proposed tests are: traversal (which includes the shortest path calculation relevant for this research), graph analysis, connected components, communities, centrality measures, pattern matching, graph anonymisation and some application domain generic operation.…”

The linked data benchmark council

“…Current graph and RDF benchmarks, however, do not fully attain all the desirable characteristics [5] (i.e., relevant, repeatable, fair, verifiable and economical), and sometimes neglect the particularities and requirements in RDF and graph data management [2,3,4,7] (e.g. complex graph queries over irregularly shaped and correlated data).…”

The Linked Data Benchmark Council Project

“…However, as graph databases are designed with different types of queries in mind, these relational benchmarks are largely inadequate here. There has been a study [9] about the characteristics that a Graph database benchmark should include. Among those, the use cases to be considered, the operations that they give rise to, and the experimental settings are the most important for designing such benchmarks.…”