Metadata Traces and Workload Models for Evaluating Big Storage Systems

Abad, Cristina L.; Luu, Huong; Roberts, Nathan; Lee, Kihwal; Lu, Yi; Campbell, Roy H.

doi:10.1109/ucc.2012.27

Cited by 22 publications

(7 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With respect to the storage layer and user data, there are several studies that focus particularly on HDFS [16], [17]: metadata, file access patterns create, read, write, delete, etc. Improved concurrency control through multi-versioning can improve I/O data throughput significantly under concurrency compared with HDFS, as demonstrated by BlobSeer [18].…”

Section: Related Workmentioning

confidence: 99%

Leveraging Adaptive I/O to Optimize Collective Data Shuffling Patterns for Big Data Analytics

Nicolae

Costa

Misale³

et al. 2017

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

Abstract-Big data analytics is an indispensable tool in transforming science, engineering, medicine, health-care, finance and ultimately business itself. With the explosion of data sizes and need for shorter time-to-solution, in-memory platforms such as Apache Spark gain increasing popularity. In this context, data shuffling, a particularly difficult transformation pattern, introduces important challenges. Specifically, data shuffling is a key component of complex computations that has a major impact on the overall performance and scalability. Thus, speeding up data shuffling is a critical goal. To this end, state-of-the-art solutions often rely on overlapping the data transfers with the shuffling phase. However, they employ simple mechanisms to decide how much data and where to fetch it from, which leads to sub-optimal performance and excessive auxiliary memory utilization for the purpose of prefetching. The latter aspect is a growing concern, given evidence that memory per computation unit is continuously decreasing while interconnect bandwidth is increasing. This paper contributes a novel shuffle data transfer strategy that addresses the two aforementioned dimensions by dynamically adapting the prefetching to the computation. We implemented this novel strategy in Spark, a popular in-memory data analytics framework. To demonstrate the benefits of our proposal, we run extensive experiments on an HPC cluster with large core count per node. Compared with the default Spark shuffle strategy, our proposal shows: up to 40% better performance with 50% less memory utilization for buffering and excellent weak scalability.

show abstract

Section: Related Workmentioning

confidence: 99%

Leveraging Adaptive I/O to Optimize Collective Data Shuffling Patterns for Big Data Analytics

Nicolae

Costa

Misale³

et al. 2017

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

show abstract

“…According to the storage and processing of the metadata [20] [21] in Hadoop, the Area of each data block and the number of key columns that satisfied the current Area are recorded during importation: this information is stored by generating an attached table named DB_TABLENAME_META. Here, DB_TABLENAME_META mainly contained three fields: BlockID, Area, and KNum.…”

Section: Storage and Application Of Metadatamentioning

confidence: 99%

An Efficient Distributed Data Management Method based key Columns Partition Preprocessing

Zhang²,

Li³

2015

IJDTA

View full text Add to dashboard Cite

show abstract

“…They suggest in-memory storage, potentially in form of a DSM (distributed shared memory) as an alternative to disk storage. Other studies focus particularly on HDFS [9,10]. Unlike our approach, the focus is on HDFS utilization (i.e.…”

Section: Related Workmentioning

confidence: 99%

Understanding Vertical Scalability of I/O Virtualization for MapReduce Workloads: Challenges and Opportunities

Nicolae

2014

Euro-Par 2013: Parallel Processing Workshops

View full text Add to dashboard Cite

Abstract. As the explosion of data sizes continues to push the limits of our abilities to efficiently store and process big data, next generation big data systems face multiple challenges. One such important challenge relates to the limited scalability of I/O, a determining factor in the overall performance of big data applications. Although paradigms like MapReduce have long been used to take advantage of local disks and avoid data movements over the network as much as possible, with increasing core count per node, local storage comes under increasing I/O pressure itself and prompts the need to equip nodes with multiple disks. However, given the rising need to virtualize large datacenters in order to provide a more flexible allocation and consolidation of physical resources (transforming them into public or private/hybrid clouds), the following questions arise: is it possible to take advantage of multiple local disks at virtual machine (VM) level in order to speed up big data analytics? If so, what are the best practices to achieve a high virtualized aggregated I/O throughput? This paper aims to answer these questions in the context of I/O intensive MapReduce workloads: it analyzes and characterizes their behavior under different virtualization scenarios in order to propose best practices for current approaches and speculate on future areas of improvement.

show abstract

Metadata Traces and Workload Models for Evaluating Big Storage Systems

Cited by 22 publications

References 11 publications

Leveraging Adaptive I/O to Optimize Collective Data Shuffling Patterns for Big Data Analytics

Leveraging Adaptive I/O to Optimize Collective Data Shuffling Patterns for Big Data Analytics

An Efficient Distributed Data Management Method based key Columns Partition Preprocessing

Understanding Vertical Scalability of I/O Virtualization for MapReduce Workloads: Challenges and Opportunities

Contact Info

Product

Resources

About