Erasure coding for small objects in in-memory KV storage

Abstract

We present MemEC, an erasure-coding-based inmemory key-value (KV) store that achieves high availability and fast recovery while keeping low data redundancy across storage servers. MemEC is specifically designed for workloads dominated by small objects. By encoding objects in entirety, MemEC is shown to incur 60% less storage redundancy for small objects than existing replication-and erasure-coding-based approaches. It also supports graceful transitions between decentralized requests in normal mode (i.e., no failures) and coordinated requests in degraded mode (i.e., with failures). We evaluate our MemEC prototype via testbed experiments under read-heavy and update-heavy YCSB workloads. We show that MemEC achieves high throughput and low latency in both normal and degraded modes, and supports fast transitions between the two modes.

Incremental Checkpointing for Fault-Tolerant Stream Processing Systems: A Data Structure Approach

“…Data replication and data persistence were two methods to keep data safe for in-memory databases such as Redis [22], [23], Memcached [24], MongoDB [25] and so on. [26]- [30] adopted data replication techniques to achieve fault tolerance. In [26], a standby replicated node would take over the role of the primary node in the event of a failure.…”

“…A transaction with the corresponding column of the in-memory database was added to the value identifier vector and the dictionary log of the slave node. [30] proposed MemEC, which was an erasure-coding based key-value store. MemEC encoded objects including keys, values, and metadata through a new fully encoded data model to achieve low data access latency and fast recovery from in-memory storage servers.…”

Análisis del rendimiento de la paralelización del algoritmo Reed-Solomon

“…La paralelización se realiza en función del número de hilos para ejecutar el proceso de codificación y decodificación que se detalla en el siguiente apartado. Según la literatura, los algoritmos de código de borrado son aplicados para almacenamiento distribuido fiable [8], memoria tolerante a fallas [9] y reconstrucción de contenido a partir de datos ampliamente distribuidos [10], [11], [12].…”