Grid Datafarm Architecture for Petascale Data Intensive Computing

Tatebe, Osamu; Morita, Y.; Matsuoka, Satoshi; Soda, Noriyuki; Sekiguchi, Satoshi

doi:10.1109/ccgrid.2002.1017117

Cited by 108 publications

(67 citation statements)

References 5 publications

(5 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…If parallelism between heterogeneous networks and elements is used, it is possible to asynchronously send the suitable block size BS C S to each resource. The block size of each element is obtained independently by means of (6). In this sense, faster storage elements receive more blocks according to their transfer rate.…”

Section: Decision-making For File Creating Requestsmentioning

confidence: 99%

“…The broker not only performs the decision-making but is also essential for managing these systems. In this sense, GAS has been conceived as a brokering-based solution and can therefore be used with minor changes in conjunction with any data access service, such as Storage Resource Manager (SRM) [5], Grid Datafarm (Gfarm) [6], MAPFS-Grid [7], etc. designed to provide self-management features to the system.…”

Section: Gas Architecturementioning

confidence: 99%

See 1 more Smart Citation

An autonomic framework for enhancing the quality of data grid services

Sánchez

Montes

Pérez

et al. 2012

Future Generation Computer Systems

View full text Add to dashboard Cite

Data grid services have been used to deal with the increasing needs of applications in terms of data volume and throughput. The large scale, heterogeneity and dynamism of grid environments often make management and tuning of these data services very complex. Furthermore, current high-performance I/O approaches are characterized by their high complexity and specific features that usually require specialized administrator skills. Autonomic computing can help manage this complexity. The present paper describes an autonomic subsystem intended to provide self-management features aimed at efficiently reducing the I/O problem in a grid environment, thereby enhancing the quality of service (QoS) of data access and storage services in the grid.Our proposal takes into account that data produced in an I/O system is not usually immediately required. Therefore, performance improvements are related not only to current but also to any future I/O access, as the actual data access usually occurs later on. Nevertheless, the exact time of the next I/O operations is unknown. Thus, our approach proposes a long-term prediction designed to forecast the future workload of grid components. This enables the autonomic subsystem to determine the optimal data placement to improve both current and future I/O operations.

show abstract

Section: Decision-making For File Creating Requestsmentioning

confidence: 99%

Section: Gas Architecturementioning

confidence: 99%

An autonomic framework for enhancing the quality of data grid services

Sánchez

Montes

Pérez

et al. 2012

Future Generation Computer Systems

View full text Add to dashboard Cite

show abstract

“…A very large distributed storage space is thus made available to applications that usually use file storage, with no need for modifications. This approach has been taken by a few projects like GFarm [34], GridNFS [13], GPFS [29], XtreemFS [14], etc. Implementing transparent access at a global scale naturally leads however to a number of challenges related to scalability and performance, as the file system is put under pressure by a very large number of concurrent accesses.…”

Section: Related Workmentioning

confidence: 99%

BlobSeer: Next-generation data management for large scale infrastructures

Nicolae

Antoniu

Bougé

et al. 2011

Journal of Parallel and Distributed Computing

106

View full text Add to dashboard Cite

As data volumes increase at a high speed in more and more application fields of science, engineering, information services, etc., the challenges posed by data-intensive computing gain an increasing importance. The emergence of highly scalable infrastructures, e.g. for cloud computing and for petascale computing and beyond introduces additional issues for which scalable data management becomes an immediate need. This paper brings several contributions. First, it proposes a set of principles for designing highly scalable distributed storage systems that are optimized for heavy data access concurrency. In particular, we highlight the potentially large benefits of using versioning in this context. Second, based on these principles, we propose a set of versioning algorithms, both for data and metadata, that enable a high throughput under concurrency. Finally, we implement and evaluate these algorithms in the BlobSeer prototype, that we integrate as a storage backend in the Hadoop MapReduce framework. We perform extensive microbenchmarks as well as experiments with real MapReduce applications: they demonstrate that applying the principles defended in our approach brings substantial benefits to data intensive applications.

show abstract

“…As such, Plan 9 nodes interoperate only with one other. In the Grid context, the construction of a single, uniform and hierarchical file namespace has been studied in [34] and [35].…”

Section: Related Workmentioning

confidence: 99%

Global-scale peer-to-peer file services with DFS

Chazapis

Tsoukalas

Verigakis

et al. 2007

2007 8th IEEE/ACM International Conference on Grid Computing

View full text Add to dashboard Cite

Abstract-The global inter-networking infrastructure that has become essential for contemporary day-to-day computing and communication tasks, has also enabled the deployment of several large-scale data sharing overlays. Communities collaboratively aggregate and distribute file and storage resources either in the controlled environment of the Grid, or hidden under the anonymity cloak created by peer-to-peer protocols. Both designs exhibit unique properties and characteristics: Peer-to-peer algorithms address the formation of vast, heterogeneous and dynamic sharing networks, while Grids focus on policy enforcement and accounting features. A distributed data management facility that will assimilate respective practices has been envisioned by numerous related research initiatives, especially when there is a need to incorporate disperse resources in large pools, without relinquishing participants of their respective rights. In this paper, we describe the Distributed File Services (DFS) architecture -a peer-to-peer service overlay, which allows distinct administrative entities to form arbitrary file distribution relationships. Each DFS peer can be uniquely authenticated and maintains direct control of its own namespace and storage assets by defining corresponding authorization directives and policies. The peerto-peer nature of the system allows for scalable deployment and resource allocation, either in a stand-alone scenario or in the Grid context. Moreover, we introduce the notion of a "web of files", as a non-hierarchical, global-scale namespace of distributed data collections and elaborate on a prototype implementation that features novel semantics for integrating our architectural principles and concepts into the operating system level.

show abstract

Grid Datafarm Architecture for Petascale Data Intensive Computing

Cited by 108 publications

References 5 publications

An autonomic framework for enhancing the quality of data grid services

An autonomic framework for enhancing the quality of data grid services

BlobSeer: Next-generation data management for large scale infrastructures

Global-scale peer-to-peer file services with DFS

Contact Info

Product

Resources

About