Compositionally-Restricted Attention-Based Network for Materials Property Prediction

Abstract

<div>In this paper, we evaluate an attention-based neural network architecture for the prediction of inorganic materials properties given access to nothing but each materials' chemical composition. We demonstrate that this novel application of self-attention for material property predictions strikingly outperforms both statistical and ensemble machine learning methods, as well as a fully-connected neural network.This Compositionally-Restricted Attention-Based network, referred to as CrabNet, is associated with improved test metrics across six of seven different tested materials properties from the AFLOW database. Moreover, we show that CrabNet outperforms other methods in the absence of chemical information, even when the statistical and ensemble learning techniques are given domain-specific chemical knowledge about the materials. Given its impressive improvement in predictive accuracy compared to previous methods, as well as its minimal hardware requirements for training and prediction, we feel confident that CrabNet, and the ideas explored within, will be central for future materials informatics research.</div>

Optimizing Fractional Compositions to Achieve Extraordinary Properties

“…A pretrained network was selected that provides accurate predictions across many different properties. For this implementation, the CrabNet architecture from Wang et al [16] was chosen. This model provides state-of-theart performance in predicting from elemental information and includes self-attention mechanisms.…”

“…The element vectors are featurized according to each element's properties using traditional mat2vec, oliynyk, jarvis, or magpie embeddings. The fractions undergo an alternating sine and cosine expansion as described by Vaswani et al [19] and implemented by Wang et al [16] in CrabNet. It is important to note that log operations are applied to half of the fractional encodings to better preserve the influence of dopants on the predicted properties.…”

“…After several iterations of updates, the new representation is passed through a fully connected residual neural network. A more detailed description and discussion of CrabNet the architecture can be found in the paper from Wang et al [16].…”

“…The successes of these examples are due in large part to the expansive, and growing databases for materials properties [6][7][8][9][10][11] but also to advanced ML tools ranging from feature generation (CGCNN [12], jarvis [13], oliynyk, magpie [14]) to algorithms (RooSt [15], CrabNet [16]).…”

CrabNet for Explainable Deep Learning in Materials Science: Bridging the Gap Between Academia and Industry

“…In addition to learning element representations for a general purpose in materials science, such in the case of mat-2vec, DL methods can also learn to relate element characteristics on a material property-specific basis. For example, element embeddings were extracted from the CrabNet and HotCrab models which were reproduced using the supplied model weights and the source code [57,58]. The CrabNet and HotCrab models use mat2vec and onehot-encoded element features as the starting element representations, respectively.…”