Metadata Organization and Management for Globalization of Data Access with Onedata

Wrzeszcz, Michał; Trzepla, Krzysztof; Słota, Rafał; Zemek, Konrad; Lichoń, Tomasz; Opioła, Łukasz; Nikołow, Darin; Dutka, Łukasz; Słota, Renata; Kitowski, Jacek

doi:10.1007/978-3-319-32149-3_30

Cited by 9 publications

(7 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We found that metadata fetching may incur severe I/O overheads, especially when the file size is small and network latency is high. Figure 3 shows the metadata overhead ratios of Lustre-WAN [18], SSHFS (see Github), NFS (see Wikipedia), GVDS, and Onedata [1]. We used fio (see Wikipedia) to generate the desired types of I/O actions to evaluate the impact of file size and network latency on the metadata overhead ratio (ratio of the metadata access time and total access time).…”

Section: Background and Motivationmentioning

confidence: 99%

“…In network-based file systems (such as Onedata [1] and GFFS [2]), the client and server are decoupled and interact with each other through network communications. Several network-based file systems use client-side metadata caching to reduce the number of network communications and achieve better access performance [1][2][3][4][5]. The client caches a certain amount of metadata and periodically refreshes the cached metadata [1].…”

Section: Introductionmentioning

confidence: 99%

“…Several network-based file systems use client-side metadata caching to reduce the number of network communications and achieve better access performance [1][2][3][4][5]. The client caches a certain amount of metadata and periodically refreshes the cached metadata [1].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A self-tuning client-side metadata prefetching scheme for wide area network file systems

Wei

Xiao

Song

et al. 2021

Sci. China Inf. Sci.

View full text Add to dashboard Cite

Client-side metadata prefetching is commonly used in wide area network (WAN) file systems because it can effectively hide network latency. However, most existing prefetching approaches do not meet the various prefetching requirements of multiple workloads. They are usually optimized for only one specific workload and have no or harmful effects on other workloads. In this paper, we present a new self-tuning client-side metadata prefetching scheme that uses two different prefetching strategies and dynamically adapts to workload changes. It uses a directory-directed prefetching strategy to prefetch the related file metadata in the same directory, and a correlation-directed prefetching strategy to prefetch the related file metadata accessed across directories. A novel self-tuning mechanism is proposed to efficiently convert the prefetching strategy between directory-directed and correlation-directed prefetching. Experimental results using real system traces show that the hit ratio of the client-side cache can be significantly improved by our self-tuning client-side prefetching. With regards to the multi-workload concurrency scenario, our approach improves the hit ratios for the no-prefetching, directory-directed prefetching, variant probability graph algorithm, variant apriori algorithm, and variant semantic distance algorithm by up to 15.

show abstract

Section: Background and Motivationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A self-tuning client-side metadata prefetching scheme for wide area network file systems

Wei

Xiao

Song

et al. 2021

Sci. China Inf. Sci.

View full text Add to dashboard Cite

show abstract

“…The data-access systems use metadata to describe various bits of information connected with data access; for example, user specific information (e.g., access control) and storage-specific information (e.g., location of data replicas) [14,64]. The metadata can also be used to describe the context of data access; e.g., the load of the data-access system components.…”

Section: Roadmapmentioning

confidence: 99%

Towards Transparent Data Access With Context Awareness

Wrzeszcz

Kitowski

Słota

2018

csci

Self Cite

View full text Add to dashboard Cite

Open-data research is an important factor accelerating the production and analysis of scientific results as well as worldwide collaboration; still, very little data is being shared at scale. The aim of this article is to analyze existing data-access solutions along with their usage limitations. After analyzing the existing solutions and data-access stakeholder needs, the authors propose their own vision of a data-access model.

show abstract

“…Storage systems based on files are convenient for cluster administrators -a PFS compatible with POSIX could be mounted in Unix/Linux as a regular file system and accessed remotely when operating on files before and after computations. Modern solutions, like VeilFS [4] or OneData [20,23], are even able to hide various storage systems located in different locations behind a common interface. Another advantage of using an PFS is its integration with libraries helpful in application development.…”

Section: Introductionmentioning

confidence: 99%

Using Redis Supported by Nvram in HPC Applications

Malinowski

2017

csci

View full text Add to dashboard Cite

Nowadays, the efficiency of storage systems is a bottleneck in many modern HPC clusters. High performance in the traditional approach -processing using files -is often difficult to obtain because of a model's complexity and its read/write patterns. An alternative approach is to apply a key-value database, which usually has low latency and scales well. On the other hand, many key-value stores suffer from a limitation of memory capacity and vulnerability to serious failures, which is caused by processing in RAM. Moreover, some research suggests that scientific data models are not applicable to the storage structures of key-value databases. In this paper, the author proposes a resolution to the above-mentioned issues by replacing RAM with NVRAM. A practical example is based on Redis NoSQL. The article also contains three domain-specific APIs that show the idea behind transforming from an HPC data model to Redis structures as well as two micro-benchmarks results.

show abstract

Metadata Organization and Management for Globalization of Data Access with Onedata

Cited by 9 publications

References 7 publications

A self-tuning client-side metadata prefetching scheme for wide area network file systems

A self-tuning client-side metadata prefetching scheme for wide area network file systems

Towards Transparent Data Access With Context Awareness

Using Redis Supported by Nvram in HPC Applications

Contact Info

Product

Resources

About