Quickly generating billion-record synthetic databases

Abstract

Evaluating database system performance often requires generating synthetic databases -ones having certain statistical properties but filled with dummy information. When evaluating different database designs, it is often necessary to generate several databases and evaluate each design. As database sizes grow to terabytes, generation often takes longer than evaluation. This paper presents several database generation techniques. In particular it discusses:(1) Parallelism to get generation speedup and scaleup.(2) Congruential generators to get dense unique uniform distributions.(3) Special-case discrete logarithms to generate indices concurrent to the base table generation.(4) Modification of (2) to get exponential, normal, and self-similar distributions.The discussion is in terms of generating billion-record SQL databases using C programs running on a shared-nothing computer system consisting of a hundred processors, with a thousand discs. The ideas apply to smaller databases, but large databases present the more difficult problems.

Data Streams: Algorithms and Applications

“…We in the computer science community have traditionally focused on scaling in size: how to efficiently manipulate large disk-bound data via suitable data structures [213], how to scale to databases of petabytes [114], synthesize massive data sets [115], etc. However, far less attention has been given to benchmarking, studying performance of systems under rapid updates with near-real time analyses.…”

Crowdsourced enumeration queries

“…We need to incorporate both the true data via r i /W as well as our most pessimistic belief of the underlying skew. As a pessimistic prior, we choose the highly skewed Grays selfsimilar distribution [20], often used for the 80/20 rule. Only if we find a sequence which can not be explained (with more than 1% chance) with the 80/20 distribution, we believe we have encountered list walking.…”

A Data Generator for Cloud-Scale Benchmarking

“…An important milestone was the paper by Gray et al [12], the authors showed how to generate data sets with different distributions and dense unique sequences in linear time and in parallel. Fast, parallel generation of data with special distribution characteristics is the foundation of our data generation approach.…”