Grid-Based Data Stream Processing in e-Science

Kuntschke,; Scholl, H.; Huber,; Kemper,; Reiser,; Adorf,; Lemson,; Voges,

doi:10.1109/e-science.2006.261114

Cited by 12 publications

(13 citation statements)

References 9 publications

(4 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Within that common effort, our main focus are applications that access scientific databases from the Grid or use Grid-based data stream management [9]. While our framework is also applicable to other domains, we concentrate on data-intensive tasks from astrophysics for illustration.…”

Section: Motivationmentioning

confidence: 99%

See 1 more Smart Citation

Community Training: Partitioning Schemes in Good Shape for Federated Data Grids

Scholl¹,

Kuntschke²,

Reiser³

et al. 2007

Third IEEE International Conference on E-Science and Grid Computing (E-Science 2007)

Self Cite

View full text Add to dashboard Cite

In federated Data Grids, individual institutions share their data sets within a community to enable collaborative data analysis. Data access needs to be provided in a scalable fashion since in most e-science communities, data sets do not only grow exponentially but also experience an increasing popularity. If data autonomy is retained, each individual institution has to ensure efficient access to its data. Analyzing application-specific data properties (such as data skew) or query characteristics (query patterns) and distributing data within Data Grids accordingly, allows for improved throughput for data-intensive applications and enables better load-balancing between shared resources. We propose a framework for investigating application-specific index structures for creating suitable partitioning schemes. We evaluate two variants of the well-known Quadtree data structure as well as the Zones approach, an index structure from the astrophysics domain, according to several criteria. Our framework improves data access within federated Data Grids and can be combined with well-established Grid methods as well as with more flexible P2P technologies.

show abstract

Section: Motivationmentioning

confidence: 99%

“…Decentralized Grid-based data stream management [9] is a different approach to increase the scalability of e-science research efforts. In this approach, complex data-intensive workflows can benefit from the data stream sharing [8,10] optimization technique which comprises in-network query processing and multi-query optimization.…”

Section: Related Workmentioning

confidence: 99%

Community Training: Partitioning Schemes in Good Shape for Federated Data Grids

Scholl¹,

Kuntschke²,

Reiser³

et al. 2007

Third IEEE International Conference on E-Science and Grid Computing (E-Science 2007)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Processing of multiple data streams in grid-based peer-to-peer (P2P) networks is described in [25]. Spatial matching, a current issue in astrophysics as a real-life e-Science scenario, is introduced to show how a data stream management system (DSMS) can help in efficiently performing associated tasks.…”

Section: Aqp and The Gridmentioning

confidence: 99%

Adaptive Query Processing in Data Grids

Zhao

Cao

et al. 2010

Handbook of Research on P2P and Grid Systems for Service-Oriented Computing

View full text Add to dashboard Cite

The data grid integrates wide-area autonomous data sources and provides users with a unified data query and processing infrastructure. Adapt data query and processing is required by data grids to provide better quality of services (QoS) to users and applications in spite of dynamically changing resources and environments. Existing AQP techniques can only meet partially data grid requirements. Some existing work is either addressing domain-specific or single-node query processing problems. Data grids provide new mechanisms for monitoring and discovering data and resources in a cross-domain wide area. Data query in grids can benefit from these information and provide better adaptability to the dynamic nature of the grid environment.In this work, an adaptive controller is proposed that dynamically adjusts resource shares to multiple data query requests in order to meet a specified level of service differentiation. The controller parameters are automatically tuned at runtime based on a predefined cost function and an online learning method. Simulation results show that our controller can meet given QoS differentiation targets and adapt to dynamic system resources among multiple data query processing requests while total demand from users and applications exceeds system capability.

show abstract

“…The correlation and combination of observational data or data gained from scientific simulations (e. g., covering different wave bands) is the key for gaining new scientific insights. The creation of likelihood maps for galaxy clusters [21] or the classification of spectral energy distributions [9] are examples for such applications. The Ph.D. project described in this paper is conducted in the context of AstroGrid-D [4], the astrophysics community project within the German e-science and Grid Computing initiative D-Grid.…”

Section: Application Scenariomentioning

confidence: 99%

Community-driven data grids

Scholl

Kemper

2008

Proc. VLDB Endow.

Self Cite

View full text Add to dashboard Cite

Beyond already existing huge data volumes, e-science communities face major challenges in managing the anticipated data deluge of forthcoming projects. Community-driven data grids target at domain-specific federations and provide a distributed, collaborative data management by employing dominant data characteristics (e. g., data skew) and query patterns to optimize the overall throughput. By combining well-established techniques for data partitioning and replication with Peer-to-Peer (P2P) technologies we can address several challenging problems: data load balancing, handling of query hot spots, and the adaption to short-term burst as well as long-term load redistributions.

show abstract

Grid-Based Data Stream Processing in e-Science

Cited by 12 publications

References 9 publications

Community Training: Partitioning Schemes in Good Shape for Federated Data Grids

Community Training: Partitioning Schemes in Good Shape for Federated Data Grids

Adaptive Query Processing in Data Grids

Community-driven data grids

Contact Info

Product

Resources

About