Experiments or simulation? A characterization of evaluation methods for in-memory databases

Abstract

Abstract-The recent growth of interest for in-memory databases poses the question on whether established prediction methods such as response surfaces and simulation are effective to describe the performance of these systems. In particular, the limited dependence of in-memory technologies on the disk makes methods such as simulation more appealing than in the past, since disks are difficult to simulate. To answer this question, we study an in-memory commercial solution, SAP HANA, deployed on a high-end server with 120 physical cores. First, we apply experimental design methods to generate response surfaces that describe database performance as a function of workload and hardware parameters. Next, we develop a class-switching queueing network model to predict in-memory database performance under similar scenarios. By comparing the applicability of the two approaches to modeling multi-tenancy, we find that both queueing and response surface models yield mean prediction errors in the range 5%-22% with respect to mean memory occupancy and response times, but the accuracy for the latter deteriorates in response surfaces as the number of experiments are reduced, whereas simulation is effective in all cases. This suggests that simulation can be very effective in performance prediction for in-memory database management.

Efficient Memory Occupancy Models for In-memory Databases

“…A closer look at this scenario over time in Figure 2 also reveals that generally memory occupancy follows the fluctuation of queue lengths. We observed a similar behavior for the experiments conducted in [4].…”

“…For this analysis we looked at several different experiments. Due to space limitations we only show the results for one representative experiment during which we ran the TPC-H benchmark at scale factor 300 (equal to ≈100 GB data compressed in main-memory) with 40 concurrent users on the 8-socket system introduced in [4]. We show the results for this experiment in Figure 1.…”

“…The per-class queue lengths can then be generated using above distribution and be fed into our memory model in [4] to approximate the distribution of memory occupancy. We will confirm empirically, that this approach leads to good results across our set of experiments.…”

“…Furthermore, literature with queueing theoretic background does not insist on the explicit memory modeling part either. While we have already explored memory occupancy under multi-tenant workloads in [4], the modeling part was based on simulation. With simulation, probabilistic measures are easier to obtain than with analytical models, but this comes at an increased cost of efficiency.…”

Iris

“…Response Surface Methodology (RSM) [11,10] is a technique to identify the independent variable settings at which the response variable attains the optimum, i.e., minimum or maximum, value. An experiment design phase is first carried out to establish an initial model of the response variable as a function of the independent variables.…”