One of the largest obstacles facing scanning probe microscopy is the constant need to correct flaws in the scanning probe in situ. This is currently a manual, time-consuming process that would benefit greatly from automation. Here we introduce a convolutional neural network protocol that enables automated recognition of a variety of desirable and undesirable scanning probe tip states on both metal and non-metal surfaces. By combining the best performing models into majority voting ensembles, we find that the desirable states of H:Si(100) can be distinguished with a mean precision of 0.89 and an average receiver-operator-characteristic curve area of 0.95. More generally, high and low-quality tips can be distinguished with a mean precision of 0.96 and near perfect area-under-curve of 0.98. With trivial modifications, we also successfully automatically identify undesirable, non-surface-specific states on surfaces of Au(111) and Cu(111). In these cases we find mean precisions of 0.95 and 0.75 and area-under-curves of 0.98 and 0.94, respectively.

Currently, researchers spend significant time manually searching through large volumes of data produced during scanning probe imaging to identify specific patterns and motifs formed via self-assembly and self-organization. Here, we use a combination of Monte Carlo simulations, general statistics, and machine learning to automatically distinguish several spatially correlated patterns in a mixed, highly varied data set of real AFM images of self-organized nanoparticles. We do this regardless of feature-scale and without the need for manually labeled training data. Provided that the structures of interest can be simulated, the strategy and protocols we describe can be easily adapted to other self-organized systems and data sets.

We discuss the exciting prospects for a step change in our ability to map and modify matter at the atomic/molecular level by embedding machine learning algorithms in scanning probe microscopy (with a particular focus on scanning tunnelling microscopy, STM). This nano-AI hybrid approach has the far-reaching potential to realise a technology capable of the automated analysis, actuation, and assembly of matter with a precision down to the single chemical bond limit.

Scanning probe microscopists generally do not rely on complete images to assess the quality of data acquired during a scan. Instead, assessments of the state of the tip apex, which not only determines the resolution in any scanning probe technique, but can also generate a wide array of frustrating artefacts, are carried out in real time on the basis of a few lines of an image (and, typically, their associated line profiles.) The very small number of machine learning approaches to probe microscopy published to date, however, involve classifications based on full images. Given that data acquisition is the most time-consuming task during routine tip conditioning, automated methods are thus currently extremely slow in comparison to the tried-and-trusted strategies and heuristics used routinely by probe microscopists. Here, we explore various strategies by which different STM image classes (arising from changes in the tip state) can be correctly identified from partial scans. By employing a secondary temporal network and a rolling window of a small group of individual scanlines, we find that tip assessment is possible with a small fraction of a complete image. We achieve this with little-to-no performance penalty-or, indeed, markedly improved performance in some cases-and introduce a protocol to detect the state of the tip apex in real time.

No abstract