Abstract. The rapidly increasing number of XML-related applications indicates a growing need for efficient, dynamic, and native XML support in database management systems (XDBMS). So far, both industry and academia primarily focus on benchmarking of high-level performance figures for a variety of applications, queries, or documents -frequently executed in artificial workload scenarios -and, therefore, may analyze and compare only specific or incidental behavior of the underlying systems. To cover the full XDBMS support, it is mandatory to benchmark performance-critical components bottom-up, thereby removing bottlenecks and optimizing component behavior. In this way, wrong conclusions are avoided when new techniques such as tailored XML operators, index types, or storage mappings with unfamiliar performance characteristics are used. As an experience report, we present what we have learned from benchmarking a native XDBMS and recommend certain setups to do it in a systematic and meaningful way.
MotivationThe increasing presence of XML data and XML-enabled (database) applications is raising the demand for established XML benchmarks. During the last years, a handful of ad-hoc benchmarks emerged and some of them served as basis for on-going XML research [5,29,39], thus constituting some kind of XML "standard" benchmarks. All these benchmarks address the XDBMS behavior and performance visible at the application interface (API) and fail to evaluate and compare properties of the XDBMS components involved in XQuery processing. However, the development of native XDBMSs should be test-driven for all system layers separately, as it was successfully done in the relational world, too, before such high-level benchmarks are used to confirm suitability and efficiency of an XDBMS for a given application domain.In the same way, only high-level features such as document store/retrieve and complete XQuery expressions were drawn on the comparison and adaptation of XML benchmark capabilities [21,26,31,32]. They can be often characterized as "black-box" approaches and are apparently inappropriate to analyze the internal system behavior in a detailed way. This applies to other approaches which focused on specific problems such as handling "shredding" or NULL values efficiently, too.