RDataFrame: Easy Parallel ROOT Analysis at 100 Threads

Abstract

The Physics programmes of LHC Run III and HL-LHC challenge the HEP community. The volume of data to be handled is unprecedented at every step of the data processing chain: analysis is no exception. Physicists must be provided with first-class analysis tools which are easy to use, exploit bleeding edge hardware technologies and allow to seamlessly express parallelism. This document discusses the declarative analysis engine of ROOT, RDataFrame, and gives details about how it allows to profitably exploit commodity hardware as well as high-end servers and manycore accelerators thanks to the synergy with the existing parallelised ROOT components. Real-life analyses of LHC experiments’ data expressed in terms of RDataFrame are presented, highlighting the programming model provided to express them in a concise and powerful way. The recent developments which make RDataFrame a lightweight data processing framework are described, such as callbacks and I/O capabilities. Finally, the flexibility of RDataFrame and its ability to read data formats other than ROOT’s are characterised, as an example it is discussed how RDataFrame can directly read and analyse LHCb’s raw data format MDF.

Machine Learning with ROOT/TMVA

“…The functionality of the machine learning libraries is then fully accessible for analysis. ROOT implements this feature on top of the RDataFrame [15] infrastructure with the method AsNumpy, which allows the analyst to perform computational expensive preprocessing of the data in compiled C++ code and load only the required data to memory. See figure 1 for a code example which shows the loading of data from a ROOT file to memory and subsequently pushing the data to common Python based data analysis facilities such as Pandas [16].…”

CutLang v2: Advances in a Runtime-Interpreted Analysis Description Language for HEP Data

“…The focused DSL developments for analyses are relatively new, but a DSL has been long embedded within the ROOT framework ( Brun and Rademakers, 1997 ) under the guise of TTreeFormula, TTree::Draw and TTree::Scan, which allow visual or textual representation of TTree contents for simple and quick exploratory analysis This DSL is however limited only to simple arithmetic operations, mathematical functions and basic selection criteria. Recently, ROOT developers introduced RDataFrame, a tool to process and analyze columnar datasets as a modern alternative for data analysis ( Piparo et al, 2019 ). Although RDataFrame is not a DSL itself, it implements declarative analysis by using keywords for transformations (e.g., filtering data, defining new variables) and actions (e.g., creating histograms), and is interfaced to the ROOT classes TTreeReader and TTreeDraw.…”

Coffea-casa: an analysis facility prototype

“…Column-wise data analysis [7], in which a single operation on a vector of events replaces calculations on individual events serially, is seen as a way to for the field to take advantage of vector processing units in modern CPUs, leading to significant speed-ups in throughput. Also, declarative programming paradigms [8] can make it simpler for physicists to intuitively code their analysis.…”