Data management and transfer in high-performance computational grid environments

Allcock, Bill; Bester, Joe; Bresnahan, John; Chervenak, Ann; Foster, Ian; Kesselman, Carl; Meder, Sam; Nefedova, Veronika; Quesnel, Darcy; Tuecke, Steven

doi:10.1016/s0167-8191(02)00094-7

Cited by 501 publications

(291 citation statements)

References 11 publications

Supporting

Mentioning

286

Contrasting

Unclassified

Order By: Relevance

“…The established method for data transfer in the Grid is GridFTP [7], a protocol defined by the Global Grid Forum and adopted by the majority of the existing middleware. GridFTP extends the standard FTP, including features like the Grid Security Infrastructure (GSI) [17] and third-party control and data channels.…”

Section: Transferring Filesmentioning

confidence: 99%

“…The most fundamental of them are the Data Transfer service, responsible for moving files among grid nodes (e.g., GridFTP [7]), the Replica Location service (RLS ), which keeps track of the physical locations of files and the Optimization service, which selects the best data source for each transfer in terms of completion time and manages the dynamic replica creation/deletion according to file usage statistics.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Optimizing Data Management in Grid Environments

Zissimos

Doka

Chazapis

et al. 2009

On the Move to Meaningful Internet Systems: OTM 2009

View full text Add to dashboard Cite

Abstract. Grids currently serve as platforms for numerous scientific as well as business applications that generate and access vast amounts of data. In this paper, we address the need for efficient, scalable and robust data management in Grid environments. We propose a fully decentralized and adaptive mechanism comprising of two components: A Distributed Replica Location Service (DRLS ) and a data transfer mechanism called GridTorrent. They both adopt Peer-to-Peer techniques in order to overcome performance bottlenecks and single points of failure. On one hand, DRLS ensures resilience by relying on a Byzantine-tolerant protocol and is able to handle massive concurrent requests even during node churn. On the other hand, GridTorrent allows for maximum bandwidth utilization through collaborative sharing among the various data providers and consumers. The proposed integrated architecture is completely backwards-compatible with already deployed Grids. To demonstrate these points, experiments have been conducted in LAN as well as WAN environments under various workloads. The evaluation shows that our scheme vastly outperforms the conventional mechanisms in both efficiency (up to 10 times faster) and robustness in case of failures and flash crowd instances.

show abstract

Section: Transferring Filesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Optimizing Data Management in Grid Environments

Zissimos

Doka

Chazapis

et al. 2009

On the Move to Meaningful Internet Systems: OTM 2009

View full text Add to dashboard Cite

show abstract

“…we moved to a more semantic level. This process is shown in more detail in an OntoGrid demonstration video [14].…”

Section: Querying: Exploring Datamentioning

confidence: 99%

A Semantic Data Grid for Satellite Mission Quality Analysis

Wright¹,

Sánchez-Gestido²,

Gómez-Pérez³

et al. 2008

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. The combination of Semantic Web and Grid technologies and architectures eases the development of applications that share heterogeneous resources (data and computing elements) that belong to several organisations. The Aerospace domain has an extensive and heterogeneous network of facilities and institutions, with a strong need to share both data and computational resources for complex processing tasks. One such task is monitoring and data analysis for Satellite Missions. This paper presents a Semantic Data Grid for satellite missions, where flexibility, scalability, interoperability, extensibility and efficient development have been considered the key issues to be addressed.

show abstract

“…For example, the physical file for the logical filename "inputfile.1" will be transferred automatically to the site selected for execution of the FileBreaker transformation via GridFTP [30], which provides secure, efficient data movement in Grid environments. Input files to transformations are automatically located in the Grid by searching for physical copies of a logical file in a replica location service such as RLS [31. Output files are automatically cataloged in the same location service for use in subsequent transformations and workflows.…”

Section: Workflow Generatormentioning

confidence: 99%