Ensemble learning-iterative training machine learning for uncertainty quantification and automated experiment in atom-resolved microscopy

Abstract

Deep learning has emerged as a technique of choice for rapid feature extraction across imaging disciplines, allowing rapid conversion of the data streams to spatial or spatiotemporal arrays of features of interest. However, applications of deep learning in experimental domains are often limited by the out-of-distribution drift between the experiments, where the network trained for one set of imaging conditions becomes sub-optimal for different ones. This limitation is particularly stringent in the quest to have an automated experiment setting, where retraining or transfer learning becomes impractical due to the need for human intervention and associated latencies. Here we explore the reproducibility of deep learning for feature extraction in atom-resolved electron microscopy and introduce workflows based on ensemble learning and iterative training to greatly improve feature detection. This approach allows incorporating uncertainty quantification into the deep learning analysis and also enables rapid automated experimental workflows where retraining of the network to compensate for out-of-distribution drift due to subtle change in imaging conditions is substituted for human operator or programmatic selection of networks from the ensemble. This methodology can be further applied to machine learning workflows in other imaging areas including optical and chemical imaging.

Automated and Autonomous Experiments in Electron and Scanning Probe Microscopy

“…In cases of the prior training, we note that in many cases a small deviation in the imaging parameters can result in an out-of-distribution drift for the networks, degrading their performance. This problem can be addressed either via incorporation of the ensemble learning-iterative training (ELIT) method 77 that allows selecting the network suited for experiment from a distribution, transfer, and active learning methods. This out-of-distribution drift can further be minimized by the transition from image-specific to materialsspecific descriptors.…”

“…One possible solution is utilization of the ELIT approach 77 that starts with an ensemble of models trained on simulated data (this allows avoiding manual labeling of experimental images) to identify the features present in a specific system with predictive uncertainties and then iteratively retrains the ensemble using the identified high-confidence features. An illustration of this approach is shown in Figure 5 using experimental STEM data on graphene (right panel) as an example.…”

Towards automating structural discovery in scanning transmission electron microscopy *

“…As a static data set, presented in figure 1, we chose high-angle annular dark-field images of various materials which each contain different feature lengths. The dataset includes: NiO pillars in a La:SrMnO 3 (NiO-LSMO) matrix [49][50][51], sample of BiFeO 3 (BFO) [52][53][54] and Si-containing graphene [55,56]. Each of these images contain different information of interest, such as domain walls, defects, and dopants.…”

“…Furthermore, DCNN based atom finding is robust and hence can be performed given the pretrained networks for a broad range of image sizes, sampling, signal to noise ratios, etc. The DCNN predictions can further be improved via ensemble learning and iterative training approaches [51,55]. It is important to also note that DCNN can be trained for the semantic segmentation of mesoscopic images for determination of specific microstructural elements, e.g.…”

XGBoost automates the characterisation of reversibly actuating planar-flow-casted NiTi shape memory alloy foil

“…The evaluation and quanti cation of the performance of a SMA foil is often determined by the achieved actuation angles and the amount of force it can generate during thermally excited actuation and recovery transitions. Hence, a system that can accurately and rapidly predict force and angle being generated by an actuating SMA foil under excitement, has evidently become essential [15][16][17][18][19][20]. Elimination of time-consuming physical testing was a major factor to increase the e ciency of the SMA material characterization and new SMA material discovery [21][22][23][24][25][26].…”