TýrFS: Increasing Small Files Access Performance with Dynamic Metadata Replication

Abstract

Small files are known to pose major performance challenges for file systems. Yet, such workloads are increasingly common in a number of Big Data Analytics workflows or largescale HPC simulations. These challenges are mainly caused by the common architecture of most state-of-the-art file systems needing one or multiple metadata requests before being able to read from a file. Small input file size causes the overhead of this metadata management to gain relative importance as the size of each file decreases. In this paper we propose a set of techniques leveraging consistent hashing and dynamic metadata replication to significantly reduce this metadata overhead. We implement such techniques inside a new file system named TýrFS, built as a thin layer above the Týr object store. We prove that TýrFS increases small file access performance up to one order of magnitude compared to other state-of-the-art file systems, while only causing a minimal impact on file write throughput.

A Survey on Large Scale Metadata Server for Big Data Storage

“…In fact, sometimes, a few sizes of data consumes entire memory. If few GB of data can eat up the intact storage of an MDS as addressed in [23,38,65], thus, the system cannot scale more data size. This situation occurs frequently and it is the most likely probable event in standalone MDS.…”

“…For instance, a set of file size is less than 10 MB and it can consume entire RAM space of MDS. Many research papers address the small sized file issue, for instance, HAR [38], and HAR+ [23], TyrFS [65]. The small file problem seriously affects the performance of the file system.…”

LHF: A New Archive based Approach to Accelerate Massive Small Files Access Performance in HDFS

“…However, if we constantly append additional small files to a NHAR file, each index file of NHAR will increase sharply and deteriorate the performance of reading files. Some other methods may change the architecture of HDFS, or rely on another system such as HBase [21], or propose a completely different approach [22].…”

Tiny datablock in saving Hadoop distributed file system wasted memory

“…Then, to apply the HDFS high availability principle, all of these datablocks must be replicated 3 times in total. Lastly, all of these new datablocks and their replications must send some metadata files to the namenode to complete the files storing steps [16]. In HDFS, each datablock can host only one file, irrespective of whether or not the file fits the datablock size [17].…”