Distributed data analysis with ROOT RDataFrame

Abstract

Widespread distributed processing of big datasets has been around for more than a decade now thanks to Hadoop, but only recently higher-level abstractions have been proposed for programmers to easily operate on those datasets, e.g. Spark. ROOT has joined that trend with its RDataFrame tool for declarative analysis, which currently supports local multi-threaded parallelisation. However, RDataFrame’s programming model is general enough to accommodate multiple implementations or backends: users could write their code once and execute it as-is locally or distributedly, just by selecting the corresponding backend. This abstract introduces PyRDF, a new python library developed on top of RDataFrame to seamlessly switch from local to distributed environments with no changes in the application code. In addition, PyRDF has been integrated with a service for web-based analysis, SWAN, where users can dynamically plug in new resources, as well as write, execute, monitor and debug distributed applications via an intuitive interface.

HL-LHC Analysis With ROOT: ROOT Project Input to the HL-LHC Computing Review (Stage 2)

“…During the coming years, ROOT will continue to work on providing efficient and easy to use interfaces. Examples include interfaces to ML [7], for transparent use of multi-core and GPUs [2,8], for distributed computing [9], to Python [10], and for data visualization.…”

A Serverless Engine for High Energy Physics Distributed Analysis

“…The engine relies on the distributed RDataFrame Python package [28]. This is an extension of the RDataFrame interface that wraps the computations issued by the user in their application code in a MapReduce [29] pattern.…”

EDM4hep and podio - The event data model of the Key4hep project and its implementation

“…Although podio and the EDMs are targeted more at the usage inside HEP software frameworks where the rich structure of objects and relations among them is a necessary feature, we would like to explore the possibilities of also supporting flat data formats where such information is much harder to represent. The ROOT backend writes TTrees that can be loaded into an RDataFrame [17], which can be used to gain some first experience and to collect some feedback on how the current EDM4hep can be used in such a context. We find that from a purely technical point of view it would already be possible to implement a usable analysis framework using RDataFrame on top of the usual file structure that is produced by podio.…”