A vast array of new experimental modalities have been enabled in the past several years through the development of pixelated detectors synchronized to probe scanning electronics. Such camera systems can then acquire the rich information present in the central portion of the convergent beam electron diffraction pattern as a function of probe position (4D-STEM). These 4-dimensional (or more) datasets can be readily exploited for phase contrast ptychographic imaging [1], nanoscale strain mapping [2], unit cell resolution quantitative scanning position averaged convergent beam electron diffraction [3], and more. While such detectors are now commercially available from several manufacturers with single electron sensitivity, they are typically limited to approximately 1 millisecond (1 kHz) readout times [4] while conventional integrating-detector HAADF STEM image data is acquired at approximately 10 microsecond (100 kHz) scan rates. This speed constraint places significant limits on accessible fields of view at high resolution due to sample drift, and limits in-situ acquisition to a 4D frame rate of ~1 minute.We present here the development, installation, and characterization of the 4D Camera, a CMOS Active Pixel Sensor that consists of a 576 x 576 array of 10 μm pixels [5] of a design related to the original TEAM detector [6] and an outer HAADF detector with 16 concentric quadrant diodes (Figure 1). Full-frame data from this sensor is read out at 87 kHz, digitized locally at the camera head, and sent over 96 multi-gigabit optical links to 4 Field Programmable Gate Array (FPGA) modules for image assembly, packetization, and routing. In initial tests, the sensor exhibited single electron sensitivity from at accelerating voltages from <30 keV to 300 keV, enabling electron counting methods to effectively eliminate detector readout noise. Initial data has been acquired using a structured mask cut by focused ion beam from a 50nm SiN film coated with 1000 nm of evaporated gold (Figure 2). All data will be streamed in real time via a 400 Gbps 1 km optical link to the Cori supercomputer at the National Energy Research Scientific Computing Center (NERSC), which will perform the 4-dimensional reconstruction and HDF5 file writing before additional asynchronous processing and analysis. By design this is a parallel computational workflow, and NERSC's HPC provides concurrency and a rich software environment to scale up analysis and feedback codes. In-hardware edge-computing on these FPGA devices may also be used to carry out initial data processing (e.g. gain and dark correction, thresholding) before the data is placed on the network.
Increasingly, scientific advances require the fusion of large amounts of complex data with extraordinary amounts of computational power. The problems of deep science demand deep computing and deep storage resources. In addition to teraflop-range computing engines with their own local storage, facilities must provide large data repositories of the order of 10-100 petabytes, and networking to allow the movement of multi-terabyte files in a timely and secure manner. This paper examines such problems and identifies associated challenges. The paper discusses some of the storage systems and data management methods that are needed for computing facilities to address the challenges and describes some ongoing improvements.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.