SpaceTransformers: Language Modeling for Space Systems

Abstract

The transformers architecture and transfer learning have radically modified the Natural Language Processing (NLP) landscape, enabling new applications in fields where open source labelled datasets are scarce. Space systems engineering is a field with limited access to large labelled corpora and a need for enhanced knowledge reuse of accumulated design data. Transformers models such as the Bidirectional Encoder Representations from Transformers (BERT) and the Robustly Optimised BERT Pretraining Approach (RoBERTa) are however trained on general corpora. To answer the need for domainspecific contextualised word embedding in the space field, we propose SpaceTransformers, a novel family of three models, SpaceBERT, SpaceRoBERTa and SpaceSciBERT, respectively further pre-trained from BERT, RoBERTa and SciBERT on our domain-specific corpus. We collect and label a new dataset of space systems concepts based on space standards. We fine-tune and compare our domain-specific models to their general counterparts on a domain-specific Concept Recognition (CR) task. Our study rightly demonstrates that the models further pre-trained on a space corpus outperform their respective baseline models in the Concept Recognition task, with SpaceRoBERTa achieving significant higher ranking overall.

Towards Language-driven Scientific AI

“…We need large models that contain representations of vast amounts of knowledge in core and progressively capilar scientific disciplines, with a grounding in world knowledge and commonsense understanding, as well as the ability to continuously acquire and update the model's beliefs as the state of the art evolves. Until now, scientific language models like SciBert (Beltagy et al, 2019), BioBert (Lee et al, 2019) or SpaceRoBERTa (Berquand et al, 2021) have tried to address the challenge of domain-specifty through additional pre-training on large amounts of scientific documents that leverage largescale open access scientific resources like OpenAire 3 , arXiv 4 , Web of Science 5 or Semantic Scholar. 6 However, only such knowledge which is statistically significant in the training data is effectively captured, limiting the usefulness of pre-trained language models as knowledge bases or reasoning engines.…”