CollagenTransformer: End-to-End Transformer Model to Predict Thermal Stability of Collagen Triple Helices Using an NLP Approach

Abstract

Collagen is one of the most important structural proteins in biology, and its structural hierarchy plays a crucial role in many mechanically important biomaterials. Here, we demonstrate how transformer models can be used to predict, directly from the primary amino acid sequence, the thermal stability of collagen triple helices, measured via the melting temperature T m. We report two distinct transformer architectures to compare performance. First, we train a small transformer model from scratch, using our collagen data set featuring only 633 sequence-to-T m pairings. Second, we use a large pretrained transformer model, ProtBERT, and fine-tune it for a particular downstream task by utilizing sequence-to-T m pairings, using a deep convolutional network to translate natural language processing BERT embeddings into required features. Both the small transformer model and the fine-tuned ProtBERT model have similar R 2 values of test data (R 2 = 0.84 vs 0.79, respectively), but the ProtBERT is a much larger pretrained model that may not always be applicable for other biological or biomaterials questions. Specifically, we show that the small transformer model requires only 0.026% of the number of parameters compared to the much larger model but reaches almost the same accuracy for the test set. We compare the performance of both models against 71 newly published sequences for which T m has been obtained as a validation set and find reasonable agreement, with ProtBERT outperforming the small transformer model. The results presented here are, to our best knowledge, the first demonstration of the use of transformer models for relatively small data sets and for the prediction of specific biophysical properties of interest. We anticipate that the work presented here serves as a starting point for transformer models to be applied to other biophysical problems.

Predicting Collagen Triple Helix Stability through Additive Effects of Terminal Residues and Caps

“… The development of computational models for the prediction of T m values of collagen triple helices is currently an active area of research. [ 65 , 66 , 67 , 68 , 69 , 70 , 71 ] Several of these models succeeded in capturing the influence of inner residues on triple helix stability. These algorithms, however, do not take frameshifts, terminal capping groups and their charge into account.…”

Predicting Collagen Triple Helix Stability through Additive Effects of Terminal Residues and Caps

“…(3) The development of computational models for the prediction of T m values of collagen triple helices is currently an active area of research. [65][66][67][68][69][70][71] Several of these models succeeded in capturing the influence of inner residues on triple helix stability. These algorithms, however, do not take frameshifts, terminal capping groups and their charge into account.…”

“…Cross‐comparisons between different works have therefore been virtually impossible. Our set of parameters allows for recalculating reported T m data to a common reference peptide and, thus, enables the comparison of results from diverse sources. The development of computational models for the prediction of T m values of collagen triple helices is currently an active area of research [65–71] . Several of these models succeeded in capturing the influence of inner residues on triple helix stability.…”

“…The development of computational models for the prediction of T m values of collagen triple helices is currently an active area of research [65–71] . Several of these models succeeded in capturing the influence of inner residues on triple helix stability.…”

Computational Design and Manufacturing of Sustainable Materials through First-Principles and Materiomics

“…“Attention” mechanisms, which direct neural networks to focus on the most relevant components of input data based on context rather than sequential order, improve training efficiency and performance in RNNs. Transformer models extend attention mechanisms to “self-attention” by removing sequential recurrent processing steps altogether and processing input tokens in parallel, thus shifting training toward semisupervised and unsupervised methods. , Transformer neural networks are the current state-of-the-art in natural language processing (NLP) tasks and have recently been extended to predict chemical and physical properties of materials. − All supervised learning models are generally trained by assigning a loss function based on model performance and modifying trainable parameters through multiple iterations to minimize this loss. Artificial neural networks utilize back-propagation and gradient descent algorithms to optimally modify model parameters in each iteration.…”