Using Redis Supported by Nvram in HPC Applications

Abstract

Nowadays, the efficiency of storage systems is a bottleneck in many modern HPC clusters. High performance in the traditional approach -processing using files -is often difficult to obtain because of a model's complexity and its read/write patterns. An alternative approach is to apply a key-value database, which usually has low latency and scales well. On the other hand, many key-value stores suffer from a limitation of memory capacity and vulnerability to serious failures, which is caused by processing in RAM. Moreover, some research suggests that scientific data models are not applicable to the storage structures of key-value databases. In this paper, the author proposes a resolution to the above-mentioned issues by replacing RAM with NVRAM. A practical example is based on Redis NoSQL. The article also contains three domain-specific APIs that show the idea behind transforming from an HPC data model to Redis structures as well as two micro-benchmarks results.