UniBench: A Benchmark for Multi-model Database Management Systems

Abstract

Unlike traditional database management systems which are organized around a single data model, a multi-model database (MMDB) utilizes a single, integrated back-end to support multiple data models, such as document, graph, relational, and key-value. As more and more platforms are proposed to deal with multi-model data, it becomes crucial to establish a benchmark for evaluating the performance and usability of MMDBs. Previous benchmarks, however, are inadequate for such scenario because they lack a comprehensive consideration for multiple models of data. In this paper, we present a benchmark, called UniBench, with the goal of facilitating a holistic and rigorous evaluation of MMDBs. UniBench consists of a mixed data model, a synthetic multi-model data generator, and a set of core workloads. Specifically, the data model simulates an emerging application: Social Commerce, a Web-based application combining E-commerce and social media. The data generator provides diverse data format including JSON, XML, key-value, tabular, and graph. The workloads are comprised of a set of multi-model queries and transactions, aiming to cover essential aspects of multi-model data management. We implemented all workloads on ArangoDB and OrientDB to illustrate the feasibility of our proposed benchmarking system and show the learned lessons through the evaluation of these two multi-model databases. The source code and data of this benchmark can be downloaded at http://udbms.cs.helsinki.fi/bench/.

Holistic evaluation in multi-model databases benchmarking

“…In each phase, we generate transaction data according to the interests of customers and unite all three portions as an integral part of the multi-model dataset. The detailed implementation of the algorithm can be found in the preliminary version of this paper [51]. Next, we discuss the three phases in detail as follows:…”

“…Additional details (e.g. the detailed description, involved data models, the input and output data) can be found in the conference version of this paper [51].…”

“…We ran two individual transactions (i.e., New Order and New Payment), and a mixed transaction combining them on the SUT. The operations of transactions in detail can be found in [51]. Three DBs manage to roll back invalid transactions and commit valid ones, which means ACID properties on two multi-model transactions are guaranteed.…”

“…This article is an extension from our previous conference version [51]. In particular, this article improves the previous version mainly in four aspects: (1) introduced a new problem of parameter curation for benchmarking the multi-model query; (2) proposed a sampling-based algorithm to select the parameters with the diversity guarantee; (3) provided more theoretical results and analysis for UniBench ; (4) conducted extensive experiments over four state-of-the-art MMDBs and give an insightful analysis.…”

On Complex Value Relations in Hive

“…Queries over complex types TPC-H 11 , TPC-DS 11 , Hive-600 [14] HiBench [9], SmartBench 9 Pavlo et al [13] Hive-testbench 10 BigBench [8], TPC-xbb 11 UniBench [19] over, despite the richness of the HiveQL query language, authors of queries seem to restrict themselves to a chosen few operators over complex types.…”