The language of proteins: NLP, machine learning & protein sequences

Abstract

Natural language processing (NLP) is a field of computer science concerned with automated text and language analysis. In recent years, following a series of breakthroughs in deep and machine learning, NLP methods have shown overwhelming progress. Here, we review the success, promise and pitfalls of applying NLP algorithms to the study of proteins. Proteins, which can be represented as strings of amino-acid letters, are a natural fit to many NLP methods. We explore the conceptual similarities and differences between proteins and language, and review a range of protein-related tasks amenable to machine learning. We present methods for encoding the information of proteins as text and analyzing it with NLP methods, reviewing classic concepts such as bag-of-words, k-mers/n-grams and text search, as well as modern techniques such as word embedding, contextualized embedding, deep learning and neural language models. In particular, we focus on recent innovations such as masked language modeling, self-supervised learning and attention-based models. Finally, we discuss trends and challenges in the intersection of NLP and protein research.

Embeddings from protein language models predict conservation and variant effects

“…Every year, algorithms improve natural language processing (NLP), in particular by feeding large text corpora into Deep Learning (DL)-based Language Models (LMs). These advances have been transferred to protein sequences by learning to predict masked or missing amino acids using large databases of raw protein sequences as input (Alley et al 2019 ; Bepler and Berger 2019a , 2021 ; Elnaggar et al 2021 ; Heinzinger et al 2019 ; Madani et al 2020 ; Ofer et al 2021 ; Rao et al 2020 ; Rives et al 2021 ). Processing the information learned by such protein LMs (pLMs), e.g., by constructing 1024-dimensional vectors of the last hidden layers, yields a representation of protein sequences referred to as embeddings [Fig.…”

Protein language model embeddings for fast, accurate, alignment-free protein structure prediction

“…Protein language models (pLMs) decode aspects of the language of life. In analogy to the recent leaps in Natural Language Processing (NLP), protein language models (pLMs) learn to "predict" masked amino acids given their context using no other annotation than the amino acid sequences of 10^7-10^9 proteins (Alley et al, 2019;Asgari & Mofrad, 2015;Bepler & Berger, 2019Elnaggar et al, 2021;Heinzinger et al, 2019;Madani et al, 2020;Ofer et al, 2021;Rao et al, 2019;Rives et al, 2021;Wu et al, 2021). Toward this end, NLP words/tokens correspond to amino acids, while sentences correspond to full-length proteins in the current pLMs.…”

“…Embeddings extract the information learned by the pLMs. In analogy to LMs in NLP implicitly learning grammar, pLM embeddings decode some aspects of the language of life as written in protein sequences (Heinzinger et al, 2019;Ofer et al, 2021) which suffices as exclusive input to many methods predicting aspects of protein structure and function without any further optimization of the pLM using a second step of supervised training (Alley et al, 2019;Asgari & Mofrad, 2015;Elnaggar et al, 2021;Heinzinger et al, 2019;Madani et al, 2020;Rao et al, 2019;Rives et al, 2021), or by refining the pLM through another supervised task (Bepler & Berger, 2019. Embeddings can outperform homology-based inference based on the traditional sequence comparisons optimized over five decades (Littmann, Bordin, et al, 2021;.…”

Protein Design with Deep Learning

“…As all these approaches bypass the need of an intermediate structure and use a simple sequence representation, they are not the core of this review. For further reading, we refer the reader to recent reviews [80,81].…”