DefectNet – A Deep Convolutional Neural Network for Semantic Segmentation of Crystallographic Defects in Advanced Microscopy Images

Abstract

The current practice of identifying defects in microscopy images and deriving metrics such as dislocation density and precipitates/voids diameter remains largely in the purview of human analysis. The lack of automated defect analysis for statistically meaningful quantification of a variety types of crystallographic defects is causing an increasingly large bottleneck for rational alloy design. The first and most important step of automating defect analysis is perceptual defect identification. In terms of digital image processing, semantic segmentation best emulates human recognition of defect features -it tells what defects are in an image and where they are located. In this work, we developed a novel deep convolutional neural network (CNN) model, called DefectNet, for robust and automated semantic segmentation of three crystallographic defects including line dislocations, precipitates, and voids commonly observed in structural metals and alloys [1]. Defect semantic segmentation in TEM micrographs is a challenging deep learning task due to the nature of the image itself. Unlike everyday photographs, the interpretation of image contrast in TEM micrographs is usually not straightforward; multiple contrast mechanisms often contribute to the observation of defect features. Here, we aim at resolving this image-induced challenge by optimizing the image quality. In previous work, we established an experimental protocol for a diffraction contrast imaging scanning transmission electron microscopy (DCI STEM) technique and tailored it specifically for imaging defects in popular iron-based structural alloys [2]. Thus, the DefectNet was trained on a small set of high-quality DCI STEM defect images obtained from HT-9 martensitic steels. The performance of the resulting model for each defect was assessed quantitatively by standard semantic segmentation evaluation metrics, and the resulting defect density and size measurements compared to that from a group of human experts. Figure 1 presents the DefectNet semantic segmentation predictions for the development and test sets of the three crystallographic defects. Compared to the ground truth label, the deep learning predicted defect maps show excellent resemblance. In the comparison maps color-coded by the confusion matrix at the pixel level, we can see that the majority of pixels in the prediction map are in black and turquoise and thus correctly classified as the background and the corresponding defect. Table 1 summarizes the semantic segmentation performance of the DefectNet on the test sets. The current DefectNet was trained over a limited number of labeled DCI STEM images, but it has achieved an excellent prediction performance on the test sets, with an overall averaged pixel accuracy of 94.61±1.13%, precision of 72.12±2.73%, recall of 79.22±3.27%, and region intersection over union (IU) of 61.79±2.13%, comparable to state-of-the-art deep learning semantic segmentation algorithm. The success and source of error in DCNN prediction was analyzed for each defect features in terms o...

Machine learning in electron microscopy for advanced nanocharacterization: current developments, available tools and future outlook

“…We envision that this general model will also include the identification of planar defects and dislocations, both in an atomistic nature, and in a more macroscopic or industriallyoriented basis. [71][72][73] 2.1.2 Exploratory and knowledge-revealing routines. ML is exceling in the exploration of local descriptors and knowledge extraction.…”

“…We envision that this general model will also include the identification of planar defects and dislocations, both in an atomistic nature, and in a more macroscopic or industrially-oriented basis. 71–73…”

Review: Deep Learning in Electron Microscopy

“…To improve performance, DNNs have been trained to semantically segment images [301][302][303][304][305][306][307][308] . Semantic segmentation DNNs have been developed for focused ion beam scanning electron microscopy [309][310][311] (FIB-SEM), SEM [311][312][313][314] , STEM 287,315 , and TEM 286,310,311,[316][317][318] . For example, applications of a DNN to semantic segmentation of STEM images of steel are shown in figure 3.…”

Deep Learning Segmentation of Complex Features in Atomic-Resolution Phase-Contrast Transmission Electron Microscopy Images

“…Various studies have made use of neural networks for the segmentation of images of cells, such as Akram et al (2016), Al-Kofahi et al (2018), as well as other biological datasets, such as vasculature stacks (Teikari et al, 2016), brain tumors (Dong et al, 2017), and neuron structures (Dahmen et al, 2019). Many works have introduced application specific architectures for their studies, e.g., Kassim et al (2017) and Roberts et al (2019).…”