Neural Network for Nanoscience Scanning Electron Microscope Image Recognition

Abstract

In this paper we applied transfer learning techniques for image recognition, automatic categorization, and labeling of nanoscience images obtained by scanning electron microscope (SEM). Roughly 20,000 SEM images were manually classified into 10 categories to form a labeled training set, which can be used as a reference set for future applications of deep learning enhanced algorithms in the nanoscience domain. The categories chosen spanned the range of 0-Dimensional (0D) objects such as particles, 1D nanowires and fibres, 2D films and coated surfaces, and 3D patterned surfaces such as pillars. The training set was used to retrain on the SEM dataset and to compare many convolutional neural network models (Inception-v3, Inception-v4, ResNet). We obtained compatible results by performing a feature extraction of the different models on the same dataset. We performed additional analysis of the classifier on a second test set to further investigate the results both on particular cases and from a statistical point of view. Our algorithm was able to successfully classify around 90% of a test dataset consisting of SEM images, while reduced accuracy was found in the case of images at the boundary between two categories or containing elements of multiple categories. In these cases, the image classification did not identify a predominant category with a high score. We used the statistical outcomes from testing to deploy a semi-automatic workflow able to classify and label images generated by the SEM. Finally, a separate training was performed to determine the volume fraction of coherently aligned nanowires in SEM images. The results were compared with what was obtained using the Local Gradient Orientation method. This example demonstrates the versatility and the potential of transfer learning to address specific tasks of interest in nanoscience applications.

The first annotated set of scanning electron microscopy images for nanoscience

“…Most of the methods described in detail in this section have been presented in our previous publication 2 .…”

“…The first data analysis achievement is a service offered by the NFFA-EUROPE IDRP consisting of an automatic tool for image recognition to store, classify and label SEM images: we employed a supervised machine learning algorithm using deep convolutional neural networks for recognition of SEM images 2 . In order to train the network, it was necessary to provide a labeled training set, i.e.…”

“…In this paper, we describe the datasets presented in ref. 2 , together with the methods employed to classify and validate them: the primary dataset used for neural network training (SEM dataset, Data Citation 1), a detached smaller dataset in which a hierarchical structure has been introduced (Hierarchical dataset, Data Citation 2), and two datasets which contain ~7,000 newly classified images in addition to the SEM dataset, according to different validation criteria (Majority dataset, Data Citation 3, and 100% dataset, Data Citation 4), as described in the Technical Validation section.…”

“…To guarantee the reproducibility of the convolutional neural network training results results presented in ref. 2 , we provide the reference to the original SEM dataset (Data Citation 1) used in the paper. The SEM dataset consists of 18,577 SEM images produced at the CNR-IOM (Trieste, Italy).…”

Deep learning-guided surface characterization for autonomous hydrogen lithography

“…Initial attempts to automate surface defect recognition relied on fast Fourier transforms; 25 however, the characterization of defects was limited to a few of many different species. More recently, machine learning has been applied to assist in classification and analysis of surface structures using SPM, 26,27,28,29,30 but it has yet to be applied to surface features of the H-Si(100)-2x1 surface. Here, we implement an encoderdecoder type convolutional neural network (CNN) 31,32,33 to locate and classify features on the surface.…”