Framework for Querying Distributed Objects Managed by a Grid Infrastructure

Abstract

Abstract. Queries over scientific data often imply expensive analyses of data requiring a lot of computational resources available in Grids. We are developing a customizable query processor built on top of an established Grid infrastructure, the NorduGrid middleware, and have implemented a framework for managing long running queries in Grid environment. With the framework the user does not specify the detailed job and parallelization descriptions required by NorduGrid. Instead s/he specifies queries in terms of an application-oriented schema describing contents of files managed by the Grid and accessed through wrappers. When a query is received by the system it generates NorduGrid job descriptions submitted to NorduGrid for execution. The framework considers limitations of NorduGrid. It includes a submission mechanism, a job babysitter, and a generic data exchange mechanism. The submission mechanism generates a number of jobs for parallel execution of a user query over wrapped data files. The task of the babysitter is to submit generated jobs to NorduGrid for the execution, to monitor their execution status, and to download results from the execution. The generic exchange mechanism provides a way to exchange objects through files between Grid execution nodes and user applications.

Database Oriented Grid Middlewares

“…POQSEC [19] and GridDB-Lite [20] are two examples that stand to benefit from the easiness-to-use that database declarative languages offer. POQSEC [19] performs DQP over data stored in flat files and stands on the top of NorduGrid (NG) middleware.…”

“…POQSEC [19] performs DQP over data stored in flat files and stands on the top of NorduGrid (NG) middleware. It processes scientific analyses specified as declarative SQL-like queries over data in files distributed in the Grid.…”

The design and implementation of OGSA-DQP: A service-based distributed query processor

“…Architecturally the approach in GridDB-Lite seems rather different, in that the approach seems more bottom-up -identifying pat-terns in the use of existing Grid libraries that can be captured in a declarative manner -rather than top-down -working out how an existing query language can be deployed for data and process integration in a service-oriented environment. In POQSEC [18], like GridDB and GridDB-Lite, queries expressed in a declarative language are compiled into scripts that run over an existing middleware. As such, an analogy can be drawn with Grid workflow engines such as Pegasus [15], in which abstract characterisations of application requirements are mapped onto lower level job and file descriptions for execution.…”

Architecture design of grid GIS and its applications on resource sharing

“…Typical of the work have the DAIS (database access and integration service) [1] working group built the access grid environment protocol and database middleware, such as OGSA-DAI [2], OGSA-WebDB (OGSA Web database) [3], OGSA-DQP (OGSA distributed query processing) [4]. Related working have MyGrid [5], Polar* [6], GDIS (grid data integration system) [7,8], POQSEC (parallel object query system for expensive computations) [9], CoDIMS-G (configurable data integration middleware for the grid) [10], PALADIN (pattern-based approach to large-scale dynamic information integration) [8], [11] DartGrid, SDG (scientific data grid) [12]. Most of the above grid system is about specific domain based on the static environment, did not discuss the uncertainty of spatial resources, can not preferable support the dynamic resources integration.…”