Evolution of the ROOT Tree I/O

Abstract

The ROOT TTree data format encodes hundreds of petabytes of High Energy and Nuclear Physics events. Its columnar layout drives rapid analyses, as only those parts (“branches”) that are really used in a given analysis need to be read from storage. Its unique feature is the seamless C++ integration, which allows users to directly store their event classes without explicitly defining data schemas. In this contribution, we present the status and plans of the future ROOT 7 event I/O. Along with the ROOT 7 interface modernization, we aim for robust, where possible compile-time safe C++ interfaces to read and write event data. On the performance side, we show first benchmarks using ROOT’s new experimental I/O subsystem that combines the best of TTrees with recent advances in columnar data formats. A core ingredient is a strong separation of the high-level logical data layout (C++ classes) from the low-level physical data layout (storage backed nested vectors of simple types). We show how the new, optimized physical data layout speeds up serialization and deserialization and facilitates parallel, vectorized and bulk operations. This lets ROOT I/O run optimally on the upcoming ultra-fast NVRAM storage devices, as well as file-less storage systems such as object stores.

EDM4hep and podio - The event data model of the Key4hep project and its implementation

“…We are currently investigating if and how it is possible to support such flat data formats, also in view of such usage in Key4hep. As of now, there is no clear best solution to this problem, but we are currently also starting to look into the usage of RNTuple [18] to see if it could offer benefits not only for flat data formats, but maybe also improve I/O performance for other use cases.…”

Columnar data analysis with ATLAS analysis formats

“…A compromise has to be found if both event-by-event and columnar workflows are used on the same data format. Many of the shortcomings in the ROOT file format for pure columnar reading are addressed by the RNtuple Class in ROOT 7 [15] which is another interesting option for the future.…”

AwkwardForth: accelerating Uproot with an internal DSL

“…Though motivated by the problem of ROOT deserialization, AwkwardForth is intended for the general problem of deserializing non-columnar data formats into columnar Awkward Arrays. Many file formats, such as ProtoBuf [10], Thrift [11], Avro [12], FlatBuffers [13], and JSON [14], describe data structures in a record-oriented layout, with all fields of one record contiguous with each other, while columnar formats like simple TTrees, RNTuple [15], Parquet [16], Arrow [17], and Awkward Array place all values of one field contiguous with each other before moving on to the next field. Conversions between columnar formats can be very fast, sometimes casting, rather than copying, the columns.…”