Trace: a high-throughput tomographic reconstruction engine for large-scale datasets

Abstract

BackgroundModern synchrotron light sources and detectors produce data at such scale and complexity that large-scale computation is required to unleash their full power. One of the widely used imaging techniques that generates data at tens of gigabytes per second is computed tomography (CT). Although CT experiments result in rapid data generation, the analysis and reconstruction of the collected data may require hours or even days of computation time with a medium-sized workstation, which hinders the scientific progress that relies on the results of analysis.MethodsWe present Trace, a data-intensive computing engine that we have developed to enable high-performance implementation of iterative tomographic reconstruction algorithms for parallel computers. Trace provides fine-grained reconstruction of tomography datasets using both (thread-level) shared memory and (process-level) distributed memory parallelization. Trace utilizes a special data structure called replicated reconstruction object to maximize application performance. We also present the optimizations that we apply to the replicated reconstruction objects and evaluate them using tomography datasets collected at the Advanced Photon Source. ResultsOur experimental evaluations show that our optimizations and parallelization techniques can provide 158× speedup using 32 compute nodes (384 cores) over a single-core configuration and decrease the end-to-end processing time of a large sinogram (with 4501 × 1 × 22,400 dimensions) from 12.5 h to <5 min per iteration.ConclusionThe proposed tomographic reconstruction engine can efficiently process large-scale tomographic data using many compute nodes and minimize reconstruction times.

Improving Tomographic Reconstruction from Limited Data Using Mixed-Scale Dense Convolutional Neural Networks

“…As an example, processing a reconstructed image of 1024 × 1024 pixels with a 100-layer MS-D network takes around 200 ms with an NVidia GTX 1080 GPU (NVidia, Santa Clara, CA, USA). Since MS-D networks are able to automatically adapt to each problem, we are able to use the same network hyperparameters in all experiments: each network is 100 layers deep (excluding the input and output layer), and we use equally distributed dilations d i ∈ [1,10] by setting the dilation of layer i to d i = 1 + (i mod 10). The resulting MS-D network has around 46 thousand trainable parameters, which are initialized in the same way as described in [22].…”

“…In problems with limited data, direct algorithms often produce reconstructions with insufficient quality for further analysis [9]. Iterative algorithms are typically able to produce more accurate results for limited data [9], but their high computational costs can prohibit their application in practice [10]. Furthermore, the type of prior knowledge that is exploited by an iterative algorithm limits the type of objects the algorithm can be successfully applied to.…”

Deep Learning Accelerated Light Source Experiments

“…MemXCT is a highly optimized reconstruction engine for large-scale tomography datasets [10]. In this work, we extended our efficient stream reconstruction data analysis pipeline [8,54,55] with denoising capabilities [6,56].…”

Massively parallel 3D image reconstruction

“…Extensive prior research has shown that MBIR provides higher image quality than other methods [21,26,27]. In addition, MBIR requires only one fifth of the typical X-ray dose, and thus greatly reduces data acquisition time [2]. Because of these benefits, MBIR has great potential to be adopted in next-generation imaging systems [16].…”

“…However, MBIR's improved image quality requires several orders of magnitude more computation compared to traditional methods [30,33]. Therefore, MBIR is considered impractical for many applications [2,16].…”