Parallel interactive data analysis with PROOF

Ballintijn, M.; Biskup, Marek; Brun, R.; Canal, Philippe; Feichtinger, D.; Ganis, G.; Kickinger, G.; Peters, A.; Rademakers, Fons

doi:10.1016/j.nima.2005.11.100

Cited by 24 publications

(15 citation statements)

References 2 publications

(1 reference statement)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Parallel ROOT Facility, PROOF [25], is an extension of ROOT enabling interactive analysis of large sets of ROOT files in parallel on clusters of computers or many-core machines. More generally PROOF can parallelize the class of tasks for which solutions can be formulated as a set of independent sub-tasks (embarrassingly or ideally parallel).…”

Section: Parallel Processing Using Proofmentioning

confidence: 99%

ROOT — A C++ framework for petabyte data storage, statistical analysis and visualization

Antcheva¹,

Ballintijn²,

Bellenot

et al. 2011

Computer Physics Communications

155

107

View full text Add to dashboard Cite

ROOT is an object-oriented C++ framework conceived in the high-energy physics (HEP) community, designed for storing and analyzing petabytes of data in an efficient way. Any instance of a C++ class can be stored into a ROOT file in a machine-independent compressed binary format. In ROOT the TTree object container is optimized for statistical data analysis over very large data sets by using vertical data storage techniques. These containers can span a large number of files on local disks, the web, or a number of different shared file systems. In order to analyze this data, the user can chose out of a wide set of mathematical and statistical functions, including linear algebra classes, numerical algorithms such as integration and minimization, and various methods for performing regression analysis (fitting). In particular, the RooFit package allows the user to perform complex data modeling and fitting while the RooStats library provides abstractions and implementations for advanced statistical tools. Multivariate classification methods based on machine learning techniques are available via the TMVA package. A central piece in these analysis tools are the histogram classes which provide binning of one-and multi-dimensional data. Results can be saved in high-quality graphical formats like Postscript and PDF or in bitmap formats like JPG or GIF. The result can also be stored into ROOT macros that allow a full recreation and rework of the graphics. Users typically create their analysis macros step by step, making use of the interactive C++ interpreter CINT, while running over small data samples. Once the development is finished, they can run these macros at full compiled speed over large data sets, using onthe-fly compilation, or by creating a stand-alone batch program. Finally, if processing farms are available, the user can reduce the execution time of intrinsically parallel tasks -e.g. data mining in HEP -by using PROOF, which will take care of optimally distributing the work over the available resources in a transparent way. Antcheva et al. / Computer Physics Communications 180 (2009) [2499][2500][2501][2502][2503][2504][2505][2506][2507][2508][2509][2510][2511][2512] Program summary

show abstract

Section: Parallel Processing Using Proofmentioning

confidence: 99%

ROOT — A C++ framework for petabyte data storage, statistical analysis and visualization

Antcheva¹,

Ballintijn²,

Bellenot

et al. 2011

Computer Physics Communications

155

107

View full text Add to dashboard Cite

show abstract

“…of the machines in the test has a single 250 GB hard drive connected by a SATA controller. 1 The tests were done using simulated Event Summary Data (ESD) of the ALICE experiment [7]. All the analyzed files were located on the available nodes.…”

Section: Pos(acat)022mentioning

confidence: 99%

“…The Parallel ROOT Facility (PROOF) [1] is an extension of the ROOT analysis framework [2] aimed at interactive ROOT-like analysis on a distributed system. PROOF represents an alternative to batch systems; by dynamically optimizing the use of resources it addresses in particular the case of short-and medium-duration jobs, for which the overhead from job preparation, submission and result merging may be substantial in batch systems.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

PROOF: the Parallel ROOT Facility. Scheduling and Load-balancing

Iwaszkiewicz¹

2009

Proceedings of XI International Workshop on Advanced Computing and Analysis Techniques in Physics Research — PoS(ACAT)

View full text Add to dashboard Cite

“…In order to provide parallel interactive analysis resources to the experiment community, the ALICE experiment [3] deployed a Central Analysis Facility at CERN [4] based on PROOF [5], an extension of the ROOT system [6].…”

Section: Introductionmentioning

confidence: 99%

A prototype of a dinamically expandable Virtual Analysis Facility

Bagnasco¹,

Berzano²,

Lusso³

et al. 2009

Proceedings of XII Advanced Computing and Analysis Techniques in Physics Research — PoS(ACAT08)

View full text Add to dashboard Cite

Current Grid deployments for LHC computing do not allow efficient parallel interactive processing of data. In order to allow physicists to access interactively subsets of data, for tasks like algorithm tuning or debugging before running over a full dataset, parallel Analysis Facilities based on PROOF have been deployed by the ALICE experiment at CERN and elsewhere. Whereas large Tier-1 centres can afford to build such facilities at the expense of their Grid computing clusters, this is likely not to be true for smaller Tier-2 centres. Leveraging on the virtualisation of highly performant multi-core machines, it is possible to build a fully virtual Analysis Facility on the same Worker Nodes that compose an existing Grid cluster. Using the Xen paravirtualisation hypervisor, it is then possible to dynamically move resources from the batch instance to the interactive one when needed. We present the status of the prototype being developed.

show abstract

Parallel interactive data analysis with PROOF

Cited by 24 publications

References 2 publications

ROOT — A C++ framework for petabyte data storage, statistical analysis and visualization

ROOT — A C++ framework for petabyte data storage, statistical analysis and visualization

PROOF: the Parallel ROOT Facility. Scheduling and Load-balancing

A prototype of a dinamically expandable Virtual Analysis Facility

Contact Info

Product

Resources

About