APS: adaptable prefetching scheme to different running environments for concurrent read streams in distributed file systems

Lee, Sangmin; Hyun, Soon J.; Kim, Hong-Yeon; Kim, Young-Kyun

doi:10.1007/s11227-018-2333-6

Cited by 9 publications

(2 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Adapting readahead and prefetching. Readahead and prefetching methods are both well-studied problems [23,45,69,70] and see use in distributed systems [12,14,25,48,50,51,58,75]. Many have attempted to build statistical models to optimize and tune systems [29,69,70].…”

Section: Machine Learning Libraries For Resource-constraint Systemsmentioning

confidence: 99%

KML: Using Machine Learning to Improve Storage Systems

Akgün¹,

Aydın²,

Burford³

et al. 2021

Preprint

View full text Add to dashboard Cite

Operating systems include many heuristic algorithms designed to improve overall storage performance and throughput. Because such heuristics cannot work well for all conditions and workloads, system designers resorted to exposing numerous tunable parameters to users-essentially burdening users with continually optimizing their own storage systems and applications. Storage systems are usually responsible for most latency in I/O heavy applications, so even a small overall latency improvement can be significant. Machine learning (ML) techniques promise to learn patterns, generalize from them, and enable optimal solutions that adapt to changing workloads. We propose that ML solutions become a first-class component in OSs and replace manual heuristics to optimize storage systems dynamically. In this paper, we describe our proposed ML architecture, called KML. We developed a prototype KML architecture and applied it to two problems: optimal readahead and NFS read-size values. Our experiments show that KML consumes little OS resources, adds negligible latency, and yet can learn patterns that can improve I/O throughput by as much as 2.3× or 15× for the two use cases respectively-even for complex, never-before-seen, concurrently running mixed workloads on different storage devices.

show abstract

Section: Machine Learning Libraries For Resource-constraint Systemsmentioning

confidence: 99%

KML: Using Machine Learning to Improve Storage Systems

Akgün¹,

Aydın²,

Burford³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Scholars have applied the principle of locality to improve network data prefetching efficiency. For instance, Lee et al predicted read operations on distributed file systems on the server-side and dispatched unread requests to clients [12]. Gong et al optimized the Ceph system from the perspective of random read/write by storing data in the cache only when a user sends a write request and synchronizing with the server through dirty data merging [13].…”

Section: Introductionmentioning

confidence: 99%

Efficient Large-Capacity Caching in Cloud Storage Using Skip-Gram-Based File Correlation Analysis

Xiao,

Yu,

2023

IEEE Access

View full text Add to dashboard Cite

Designing a high-capacity cache is an essential means of improving the accessibility of cloud storage. Compared with traditional data access, cloud storage data access presents new patterns, and traditional caching strategies cannot handle the prefetching and replacement of non-hot data very well. Numerous studies have shown that file correlation can optimize cloud storage's caching and prefetching strategies. However, characterizing the correlation between files from multiple dimensions is quite complex, and the difficulty of optimizing cloud storage caching using file correlation increases accordingly. Based on the above shortcomings, this study designed a file similarity strategy based on skip-gram from the analysis of user access. This strategy completes the prefetching and replacing files in a high-capacity cache by judging the correlation between files. The strategy prefetches files and dynamically inserts them into the cache by judging the correlation between files. After using the prefetching strategy, we significantly improve the cache hit rate in the simulation benchmark. In addition, the strategy can establish an index table after each training completion, which consumes very little time during online operations. During training, the time required to establish the index is O(N * log(V )), and the time for indexing is O(1).INDEX TERMS cache strategy, cloud storage, file correlation, hit rate, machine learning, prefetching.

show abstract

File fetching in distributed file system via optimization assisted hybrid deep learning model

Soundharya,

Vadivu

2024

Multimed Tools Appl

View full text Add to dashboard Cite

APS: adaptable prefetching scheme to different running environments for concurrent read streams in distributed file systems

Cited by 9 publications

References 24 publications

KML: Using Machine Learning to Improve Storage Systems

KML: Using Machine Learning to Improve Storage Systems

Efficient Large-Capacity Caching in Cloud Storage Using Skip-Gram-Based File Correlation Analysis

File fetching in distributed file system via optimization assisted hybrid deep learning model

Contact Info

Product

Resources

About