AwkwardForth: accelerating Uproot with an internal DSL

Abstract

File formats for generic data structures, such as ROOT, Avro, and Parquet, pose a problem for deserialization: it must be fast, but its code depends on the type of the data structure, not known at compile-time. Just-in-time compilation can satisfy both constraints, but we propose a more portable solution: specialized virtual machines. AwkwardForth is a Forth-driven virtual machine for deserializing data into Awkward Arrays. As a language, it is not intended for humans to write, but it loosens the coupling between Uproot and Awkward Array. AwkwardForth programs for deserializing record-oriented formats (ROOT and Avro) are about as fast as C++ ROOT and 10–80× faster than fastavro. Columnar formats (simple TTrees, RNTuple, and Parquet) only require specialization to interpret metadata and are therefore faster with precompiled code.

A comparison of the neutron detection efficiency and response characteristics of two pixelated PSD-capable organic scintillator detectors with different photo-detection readout methods

“…Some high-level analysis is performed in Python, including the time offset calibration shown in figure 5 and the neutron efficiency calculations described in section 4. The Python packages we use most heavily are NumPy [15], SciPy [16], MatPlotLib [17], and upRoot [18]. Digital waveform processing is applied to pulses from both the PMT-based and SiPM-based detectors and are then organized by time and pixel number.…”

Using a DSL to read ROOT TTrees faster in Uproot

“…AwkwardForth [4] is a DSL designed to satisfy the requirements listed in the last section. It is based on standard Forth with some additional built-in commands for parsing files.…”

“…Only a handful of libraries in Python allow users to read ROOT files, and only one allows them to do so without any compiled code dependency, that library is Uproot [2]. Uproot is a part of the Scikit-HEP ecosystem with a ROOT file reader implemented in pure Python.…”