gLite Job Provenance

Abstract

The Job Provenance (JP) service is designed to automate keeping track of computations on large scale Grids, giving thus users a tool to correctly archive information about their jobs and to re-submit any job in a reconstructed environment. JP provides a permanent minimal record of job (and its environment) related information, to which free-form user annotations can be added. JP also offers the capability of configuring any number of indexed logical views on the large collections of raw data, allowing efficient processing of even complex user queries selecting on both system data and the annotations. The scalable architecture, capable to handle millions of jobs in a single JP installation, and integrated into the EGEE gLite middleware environment is presented.

Latin American Perspectives on Grid Computing, from Bahia Blanca, Argentina

“…This is, in part, due to the fact that many of these additional standards are new, and robust and efficient implementations are only now becoming available. Globus Toolkit [5] and large projects like gLITE [4] are widely used to provide the additional foundational elements such as security and data and job management, and the domain-specific grids are building the application specific services they need on top of the foundational tools as opposed to on top of a suite of services compatible with standards. While there is substantial repetition of design and functionality from one domain-specific grid service stack to the next, there has been very little movement towards standardization with common building blocks as considered in OGF.…”

Scheduling and planning job execution of loosely coupled applications

“…Similar to AppLeS, Nimrod-G performed application specific scheduling but the process was based on deadlines and an economy model. Subsequent projects developed by the community took a different approach to job scheduling where by focusing on ease of use rather than strictly job performance [16][17][18]. Such schedulers focus on wide-spread applicability and simplicity of use by supporting the "submit and forget" methodology [19].…”

Job Provenance – Insight into Very Large Provenance Datasets

“…JP [1] was developed to keep tracks of job execution in a Grid environment. Since references to input and output datasets as well as arbitrary application-specific attributes can be easily recorded with the jobs, the records gathered by JP form a natural provenance of the datasets.…”

“…In the context of computational experiments they also serve as a source of additional information when results are out of expected bounds, grouped in new ways or organized in patterns not encountered previously. Systems like Job Provenance (JP) [1,2], which keep track of a potentially huge number of executed jobs [3,4,5], are able to either guarantee or to repute that the computational experiment itself was properly executed and correct data were used.…”