CrabNet for Explainable Deep Learning in Materials Science: Bridging the Gap Between Academia and Industry

Abstract

Despite recent breakthroughs in deep learning for materials informatics, there exists a disparity between their popularity in academic research and their limited adoption in the industry. A significant contributor to this “interpretability-adoption gap” is the prevalence of black-box models and the lack of built-in methods for model interpretation. While established methods for evaluating model performance exist, an intuitive understanding of the modeling and decision-making processes in models is nonetheless desired in many cases. In this work, we demonstrate several ways of incorporating model interpretability to the structure-agnostic Compositionally Restricted Attention-Based network, CrabNet. We show that CrabNet learns meaningful, material property-specific element representations based solely on the data with no additional supervision. These element representations can then be used to explore element identity, similarity, behavior, and interactions within different chemical environments. Chemical compounds can also be uniquely represented and examined to reveal clear structures and trends within the chemical space. Additionally, visualizations of the attention mechanism can be used in conjunction to further understand the modeling process, identify potential modeling or dataset errors, and hint at further chemical insights leading to a better understanding of the phenomena governing material properties. We feel confident that the interpretability methods introduced in this work for CrabNet will be of keen interest to materials informatics researchers as well as industrial practitioners alike.

Predicting thermoelectric transport properties from composition with attention-based deep learning

“…Third, the use of an attention-based model, in combination with the Robust L2 loss, both leads to superior performance and provides unique advantages. The learned attention weights provide an opportunity to interpret the predictions made for a composition [126], and this could be a useful aspect of using the CraTENet model when analysing individual materials (rather than in bulk, as we have focused on here). Additionally, the Robust L2 loss is especially useful in that it allows the model to learn to quantify the uncertainty arising from mapping the composition (and optionally band gap) to thermoelectric properties.…”

Machine learning enabled exploration of multicomponent metal oxides for catalyzing oxygen reduction in alkaline media

“…The symbolic regression model was built using the PySR library, 48 while the ROOST and CrabNet models were implemented using their respective open resources. [50][51][52][53] The Matplotlib 55 and Mpltern 56 tools were employed to plot the ternary plots.…”

“…We employed XGBoost, 45 LightGBM, 46 ANN, 47 symbolic regression, 48,49 deep representation learning from stoichiometry (ROOST) 50,51 and compositionally restricted attention-based networks (CrabNet) 52,53 to develop predictive models for current densities in the unit of mA cm −2 . Details can be found in the ESI.…”

“…These methods can be combined with deep learning approaches, which may offer better performance, such as ROOST 50,51 and CrabNet. 52,53 Fig. S4 † illustrates the performance of models constructed using ROOST and CrabNet on both the training and test datasets.…”

Molecular graph transformer: stepping beyond ALIGNN into long-range interactions

“…Common applications of GDL include shape analysis and pose recognition in computer vision, 1 link and community detection on social media networks, 2–4 representation learning on textual graphs, 5,6 medical image analysis for disease detection 7–9 and property prediction for molecular and crystalline materials. 10–18…”