An Evolutionary Path to Object Storage Access

Goodell, David; Kim, Seong Jo; Latham, Robert; Kandemir, Mahmut; Ross, Robert

doi:10.1109/sc.companion.2012.17

Cited by 6 publications

(3 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In many cases, however, POSIX is unnecessarily strict (Kimpe and Ross, 2014). It may hurt the system’s scalability and the ability to control the small objects contained in a file independently (Goodell et al, 2012). Object-based systems and KVStores provide better flexibility with an object-based interface, instead of the file-based interface.…”

Section: Related Workmentioning

confidence: 99%

“…Many works have compared the advantages and disadvantages of file systems and object storage systems (Brim et al, 2013; Gibson et al, 1996; Group, 2012). Other works have tried to integrate PFSs and object storage systems (Devulapalli et al, 2007) or expose the underlying object data streams of a PFS (Goodell et al, 2012) to gain better I/O performance. This study explores whether HPC workloads can benefit from an object-based system such as KVStore.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Rethinking key–value store for parallel I/O optimization

Kougkas

Eslami

Sun

et al. 2016

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

Key–value stores are being widely used as the storage system for large-scale internet services and cloud storage systems. However, they are rarely used in HPC systems, where parallel file systems are the dominant storage solution. In this study, we examine the architecture differences and performance characteristics of parallel file systems and key–value stores. We propose using key–value stores to optimize overall Input/Output (I/O) performance, especially for workloads that parallel file systems cannot handle well, such as the cases with intense data synchronization or heavy metadata operations. We conducted experiments with several synthetic benchmarks, an I/O benchmark, and a real application. We modeled the performance of these two systems using collected data from our experiments, and we provide a predictive method to identify which system offers better I/O performance given a specific workload. The results show that we can optimize the I/O performance in HPC systems by utilizing key–value stores.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Rethinking key–value store for parallel I/O optimization

Kougkas

Eslami

Sun

et al. 2016

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

show abstract

“…Recently, the "end-of-files" (EOF) [15] extension to the Parallel Virtual File System (PVFS) [7] was created to expose the object storage abstraction directly into the client space, presenting a file as a set of distinct, physically distributed objects that applications forward requests directly to, as opposed to a more implicit mapping via a striping function. Each object has an independent address space, potentially simplifying the layout of complex datasets.…”

Section: Eof Data Managementmentioning

confidence: 99%

RADAR: Runtime Asymmetric Data-Access Driven Scientific Data Replication

Jenkins

Zou

Tang

et al. 2014

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. Efficient I/O on large-scale spatiotemporal scientific data requires scrutiny of both the logical layout of the data (e.g., row-major vs. column-major) and the physical layout (e.g., distribution on parallel filesystems). For increasingly complex datasets, hand optimization is a difficult matter prone to error and not scalable to the increasing heterogeneity of analysis workloads. Given these factors, we present a partial data replication system called RADAR. We capture datatype-and collective-aware I/O access patterns (indicating logical access) via MPI-IO tracing and use a combination of coarse-grained and fine-grained performance modeling to evaluate and select optimized physical data distributions for the task at hand. Unlike conventional methods, we store all replica data and metadata, along with the original untouched data, under a single file container using the object abstraction in parallel filesystems. Our system results in manyfold improvements in some commonly used subvolume decomposition access patterns. Moreover, the modeling approach can determine whether such optimizations should be undertaken in the first place.

show abstract