The log-structured merge-tree (LSM-tree)

O’Neil, Patrick E.; Cheng, E.T.; Gawlick, Dieter; O’Neil, Elizabeth

doi:10.1007/s002360050048

Cited by 907 publications

(362 citation statements)

References 14 publications

Supporting

Mentioning

347

Contrasting

Unclassified

Order By: Relevance

“…Compared to the insert-ordered log-structured data structures, write-optimized indexes (WOIs), such as logstructured merge trees (LSM-trees) [8], stratified B-trees [9], and fractal trees [10] buffer and sort updates in memory before writing to disk. Although a bit slower than insert-ordered log-structure stores when writing, they provide a better tradeoff between update and query operations.…”

Section: Related Workmentioning

confidence: 99%

“…Compared to the conventional read-optimized indexes (ROIs, i.e., B+ Trees), recent efforts have focused on writeoptimized indexes (WOIs), such as log-structured merge trees (LSM-trees) [8], stratified B-trees [9], and fractal trees [10]. Several commercial KV stores [11]- [16] have successfully adopted WOIs, and a series of write-optimized file systems [17]- [20] has also been proposed.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

RFS: An LSM-Tree-Based File System for Enhanced Microdata Performance

Wang

Zhang

et al. 2016

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

SUMMARY File system workloads are increasing write-heavy. The growing capacity of RAM in modern nodes allows many reads to be satisfied from memory while writes must be persisted to disk. Today's sophisticated local file systems like Ext4, XFS and Btrfs optimize for reads but suffer from workloads dominated by microdata (including metadata and tiny files). In this paper we present an LSM-tree-based file system, RFS, which aims to take advantages of the write optimization of LSM-tree to provide enhanced microdata performance, while offering matching performance for large files. RFS incrementally partitions the namespace into several metadata columns on a per-directory basis, preserving disk locality for directories and reducing the write amplification of LSM-trees. A writeordered log-structured layout is used to store small files efficiently, rather than embedding the contents of small files into inodes. We also propose an optimization of global bloom filters for efficient point lookups. Experiments show our library version of RFS can handle microwrite-intensive workloads 2-10 times faster than existing solutions such as Ext4, Btrfs and XFS. key words: local file systems, log-structured merge tree, microdata, namespace partitioning, write-ordered write-optimized store

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

RFS: An LSM-Tree-Based File System for Enhanced Microdata Performance

Wang

Zhang

et al. 2016

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

show abstract

“…To support write-heavy applications, PFS borrows ideas from LevelDB [45] and utilizes Log-Structured Merge (LSM) tree [46] to reduce the I/O latency. The LSM tree offers fast random updates, inserts and deletes without sacrificing lookup performance [47].…”

Section: Write and Read In Pfsmentioning

confidence: 99%

Towards realistic benchmarking for cloud file systems: Early experiences

Ren

Shi

Wan

2014

2014 IEEE International Symposium on Workload Characterization (IISWC)

View full text Add to dashboard Cite

Abstract-Over the past few years, cloud file systems such as Google File System (GFS) and Hadoop Distributed File System (HDFS) have received a lot of research efforts to optimize their designs and implementations. A common issue for these efforts is performance benchmarking. Unfortunately, many system researchers and engineers face challenges on making a benchmark that reflects real-life workload cases, due to the complexity of cloud file systems and vagueness of I/O workload characteristics. They could easily make incorrect assumptions about their systems and workloads, leading to the benchmark results differing from the fact.As the preliminary step for designing a realistic benchmark, we make an effort to explore the characteristics of data and I/O workload in a production environment. We collected a two-week I/O workload trace from a 2,500-node production cluster, which is one of the largest cloud platforms in Asia. This cloud platform provides two public cloud services: data storage service (DSS) and data processing service (DPS). We analyze the commonalities and individualities between both cloud services in multiple perspectives, including the request arrival pattern, request size, data population and so on. Eight key observations are highlighted from the comprehensive study, including the arrival rate of requests follows a Lognormal distribution rather than a Poisson distribution, request arrival presents multiple periodicities, cloud file systems fit partly-open model rather than purely open model or closed model. Based on the comparative analysis results, we derive several interesting implications on guiding system researchers and engineers to build a realistic benchmark on their own systems. Finally, we discuss several open issues and challenges raised on benchmarking cloud file systems.

show abstract

“…When it is, the sample itself is assumed to be stored in main memory during the single pass-avoiding issues of efficiently maintaining the sample on disk. Several I/O efficient index structures such as LSM Trees [19] and Y-Trees [10] can be used to maintain a large random sample on disk. However, like geometric file, they also require frequent in-place updates, making them unsuitable for flash.…”

Section: Related Workmentioning

confidence: 99%

Online maintenance of very large random samples on flash storage

Nath

Gibbons

2009

The VLDB Journal

View full text Add to dashboard Cite

Recent advances in flash media have made it an attractive alternative for data storage in a wide spectrum of computing devices, such as embedded sensors, mobile phones, PDA's, laptops, and even servers. However, flash media has many unique characteristics that make existing data management/analytics algorithms designed for magnetic disks perform poorly with flash storage. For example, while random (page) reads are as fast as sequential reads, random (page) writes and in-place data updates are orders of magnitude slower than sequential writes. In this paper, we consider an important fundamental problem that would seem to be particularly challenging for flash storage: efficiently maintaining a very large (100 MBs or more) random sample of a data stream (e.g., of sensor readings). First, we show that previous algorithms such as reservoir sampling and geometric file are not readily adapted to flash. Second, we propose B-FILE, an energy-efficient abstraction for flash media to store self-expiring items, and show how a B-FILE can be used to efficiently maintain a large sample in flash. Our solution is simple, has a small (RAM) memory footprint, and is designed to cope with flash constraints in order to reduce latency and energy consumption. Third, we provide techniques to maintain biased samples with a B-FILE and to query the large sample stored in a B-FILE for a subsample of an arbitrary size. Finally, we present an evaluation with flash media that shows our techniques are several orders of magnitude faster and more energy-efficient than (flash-friendly versions of) reservoir sampling and geometric file. A key finding of our study, of potential use to many flash algorithms beyond sampling, is that "semi-random" writes (as defined in the paper) on flash cards are over two orders of magnitude faster and more energy-efficient than random writes.

show abstract

The log-structured merge-tree (LSM-tree)

Cited by 907 publications

References 14 publications

RFS: An LSM-Tree-Based File System for Enhanced Microdata Performance

RFS: An LSM-Tree-Based File System for Enhanced Microdata Performance

Towards realistic benchmarking for cloud file systems: Early experiences

Online maintenance of very large random samples on flash storage

Contact Info

Product

Resources

About