Increases in the data rates of detectors for electron microscopy (EM) have outpaced increases in network, mass storage and memory bandwidth by two orders of magnitude between 2009(Weber, 2018. The LiberTEM open source platform (Clausen et al., 2020) is designed to match the growing performance requirements of EM data processing (Weber, Clausen, & Dunin-Borkowski, 2020).
MotivationThe data rate of the fastest detectors for electron microscopy that are available in 2019 exceeds 50 GB/s, which is faster than the memory bandwidth of typical personal computers (PCs) at this time. Applications from ten years before that ran smoothly on a typical PC have evolved into numerical analysis of complex multidimensional datasets (Ophus, 2019) that require distributed processing on high-performance systems. Furthermore, electron microscopy is interactive and visual, and experiments performed inside electron microscopes (so-called in situ experiments) often rely on fast on-line data processing as the experimental parameters need to be adjusted based on the observation results. As a consequence, modern data processing systems for electron microscopy should be designed for very high throughput in combination with short response times for interactive GUI use and closed-loop feedback. That requires fundamental changes in the architecture and programming model, and consequently in the implementation of algorithms and user interfaces for electron microscopy applications.