File systems unfit as distributed storage backends

Abstract

For a decade, the Ceph distributed file system followed the conventional wisdom of building its storage backend on top of local file systems. This is a preferred choice for most distributed file systems today because it allows them to benefit from the convenience and maturity of battle-tested code. Ceph's experience, however, shows that this comes at a high price. First, developing a zero-overhead transaction mechanism is challenging. Second, metadata performance at the local level can significantly affect performance at the distributed level. Third, supporting emerging storage hardware is painstakingly slow. Ceph addressed these issues with BlueStore, a new backend designed to run directly on raw storage devices. In only two years since its inception, BlueStore outperformed previous established backends and is adopted by 70% of users in production. By running in user space and fully controlling the I/O stack, it has enabled space-efficient metadata and data checksums, fast overwrites of erasure-coded data, inline compression, decreased performance variability, and avoided a series of performance pitfalls of local file systems. Finally, it makes the adoption of backwards-incompatible storage hardware possible, an important trait in a changing storage landscape that is learning to embrace hardware diversity. CCS Concepts • Information systems → Distributed storage; • Software and its engineering → File systems management; Software performance.

A New Collaborative Scheduling Mechanism Based on Grading Mapping for Resource Balance in Distributed Object Cloud Storage System

“…In the traditional Ceph storage system, the PG does not have the ability to redirect. The Reweight artificial modification mechanism is an external enhancement available in the BlueStore [16]. Therefore, in order to solve the balanced storage problem, the PG redirection capability is first given, so this capability is contained within the PG.…”

“…Ceph [3] is a representative object-based storage system [4] (OBSS), which hasn't a dedicated metadata server [5] (MDS) to record the object-based storage device (OSD) locations of segmented objects. The OSD location uses a specific mapping algorithm CRUSH [6] to determine the locations of the objects and the replicas.…”

“…2, we will introduce the Ceph storage process, analyze the problems and the factors that have been missed by existing researches, and then a new mechanism is proposed in Sect. 3. The experiments to verify the proposed mechanism are given in Sect.…”

High Performance Distributed Web-Scraper

“…When a node fails, Ceph can self-heal without downtime, thereby improving system availability. Ceph offers 3 types of interface for interacting with storage, a POSIX compatible [23] file system, an S3 storage and a block device, thus providing higher compatibility with already written software.…”

Krace: Data Race Fuzzing for Kernel File Systems

“…In the current multi-core era, concurrency has been a major thrust for performance improvements, especially for system software. As is evident in kernel and file system evolutions [1][2][3][4], a whole zoo of programming paradigms is introduced to exploit multi-core computation, including but not limited to asynchronous work queues, read-copy-update (RCU), and optimistic locking such as sequence locks. However, alongside performance improvements, concurrency bugs also find their ways to the code base and have become particularly detrimental to the reliability and security of file systems due to their devastating effects such as deadlocks, kernel panics, data inconsistencies, and privilege escalations [5][6][7][8][9][10][11][12].…”