A Typical Case of Recommending the Use of a Hierarchical Data Format

Abstract

Storing large amounts of data in an efficient way from the point of view of very fast retrieval is an important requirement for many industries like KEMA, a company offering consulting, testing and certification in the energy business. Because their relational database could not cope fast enough with the large amounts of data involved, an alternative way for data storage was proposed, called Hierarchical Data Format 5 (HDF5). HDF5 is a data model, a library, and file format for storing and managing data. Four hierarchical designs for storing and retrieving the large amounts of data involved were investigated. A benchmark was carried out in order to know which hierarchical structure would perform best. Eventually, a benchmark between HDF5 and MS SQL Server was carried out. It could be shown that HDF5 performs four (4) times better for inserting and even 200 times better for retrieving data than the MS. SQL Server.