MagBERT: Magnetics Knowledge Aware Language Model Coupled with a Question Answering Pipeline for Curie Temperature Extraction Task

Abstract

In this study, we develop and release two Bidirectional Encoder Representations (BERT) models that are trained primarily with roughly ≈144 K peer-reviewed publications within the magnetic materials domain, which we refer to as MagBERT and MagMatBERT, respectively. These transformer models are then used in chemical named entity recognition (CNER) and question answering (QA) tasks in a data extraction workflow. We demonstrate this approach by developing a magnetics data set of well-known magnetic property, i.e., Curie temperature T C . We evaluate the efficacy of these models along with the ones that were not trained with magnetics corpus in CNER and QA tasks. Our results indicate that an initial training using the magnetics corpus brings about an enhancement in these tasks. Additionally, the quality of each data set is assessed by comparing them with a manually developed ground truth T C data set as well as by employing a random forest (RF) model in its predictive ability of T C as the target quantity. Our analyses demonstrate that models pretrained with magnetic corpus, i.e., MagBERT and MagMatBERT are more efficient than the ones without pretraining.