The geometry of hidden representations of protein language models

Valeriani, Lucrezia; Cuturello, Francesca; Ansuini, Alessio; Cazzaniga, Alberto

doi:10.1101/2022.10.24.513504

Cited by 10 publications

(15 citation statements)

References 25 publications

(45 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More generally, the last hidden layer has been optimized for the un-supervised pre-training objective (learning to reproduce masked sequences). This optimization might not be best for any downstream task 56,57 . Fine-tuning may exactly address this issue by optimizing the models output for particular tasks.…”

Section: Discussionmentioning

confidence: 99%

Fine-tuning protein language models boosts predictions across diverse tasks

Schmirler,

Heinzinger,

Rost

2023

Preprint

View full text Add to dashboard Cite

Prediction methods inputting embeddings from protein Language Models (pLMs) have reached or even surpassed state-of-the-art (SOTA) performance on many protein prediction tasks. In natural language processing (NLP) fine-tuning Language Models has become thede factostandard. In contrast, most pLM-based protein predictions do not back-propagate to the pLM. Here, we compared the fine-tuning of three SOTA pLMs (ESM2, ProtT5, Ankh) on eight different tasks. Two results stood out. Firstly, task-specific supervised fine-tuning almost always improved downstream predictions. Secondly, parameter-efficient fine-tuning could reach similar improvements consuming substantially fewer resources. Put simply: always fine-tune pLMs and you will mostly gain. To help you, we provided easy-to-use notebooks for parameter efficient fine-tuning of ProtT5 for per-protein (pooling) and per-residue prediction tasks athttps://github.com/agemagician/ProtTrans/tree/master/Fine-Tuning.

show abstract

Section: Discussionmentioning

confidence: 99%

Fine-tuning protein language models boosts predictions across diverse tasks

Schmirler,

Heinzinger,

Rost

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…By analyzing the intrinsic dimension (ID) in transformers, Valeriani et al (2023) discovered that data initially spans a high-dimensional manifold in the earliest layers but then undergoes considerable contraction. These insights pave the way for potentially differentiated strategies in handling manifold representations at various depths of a network.…”

Section: Discussionmentioning

confidence: 99%

Improving the Diversity of Bootstrapped DQN by Replacing Priors With Noise

et al. 2023

View full text Add to dashboard Cite

Q-learning is one of the most well-known Reinforcement Learning algorithms. There have been tremendous efforts to develop this algorithm using neural networks. Bootstrapped Deep Q-Learning Network is amongst them. It utilizes multiple neural network heads to introduce diversity into Q-learning. Diversity can sometimes be viewed as the amount of reasonable moves an agent can take at a given state, analogous to the definition of the exploration ratio in RL. Thus, the performance of Bootstrapped Deep Q-Learning Network is deeply connected with the level of diversity within the algorithm. In the original research, it was pointed out that a random prior could improve the performance of the model. In this article, we further explore the possibility of replacing priors with noise and sample the noise from a Gaussian distribution to introduce more diversity into this algorithm. We conduct our experiment on the Atari benchmark and compare our algorithm to both the original and other related algorithms. The results show that our modification of the Bootstrapped Deep Q-Learning algorithm achieves significantly higher evaluation scores across different types of Atari games. Thus, we conclude that replacing priors with noise can improve Bootstrapped Deep Q-Learning's performance by ensuring the integrity of diversities.

show abstract

“…The aim of this study is to provide researchers with a database that could accelerate the identification of novel protein families in the Human Gastrointestinal Proteome, thus potentially enabling the formulation of new functional hypotheses. Future plans include further optimization of the clustering procedure allowing for faster processing of newly deposited metagenomic data, and combining automatic classification with functional annotation leveraging deep learning 21,22 for even more comprehensive annotation of UHGP.…”

Section: Background and Summarymentioning

confidence: 99%

Protein family annotation for the Unified Human Gastrointestinal Proteome by DPCfam clustering

Barone,

Russo,

Villegas Garcia

et al. 2024

Sci Data

View full text Add to dashboard Cite

Technological advances in massively parallel sequencing have led to an exponential growth in the number of known protein sequences. Much of this growth originates from metagenomic projects producing new sequences from environmental and clinical samples. The Unified Human Gastrointestinal Proteome (UHGP) catalogue is one of the most relevant metagenomic datasets with applications ranging from medicine to biology. However, the low levels of sequence annotation may impair its usability. This work aims to produce a family classification of UHGP sequences to facilitate downstream structural and functional annotation. This is achieved through the release of the DPCfam-UHGP50 dataset containing 10,778 putative protein families generated using DPCfam clustering, an unsupervised pipeline grouping sequences into single or multi-domain architectures. DPCfam-UHGP50 considerably improves family coverage at protein and residue levels compared to the manually curated repository Pfam. In the hope that DPCfam-UHGP50 will foster future discoveries in the field of metagenomics of the human gut, we release a FAIR-compliant database of our results that is easily accessible via a searchable web server and Zenodo repository.

show abstract

The geometry of hidden representations of protein language models

Cited by 10 publications

References 25 publications

Fine-tuning protein language models boosts predictions across diverse tasks

Fine-tuning protein language models boosts predictions across diverse tasks

Improving the Diversity of Bootstrapped DQN by Replacing Priors With Noise

Protein family annotation for the Unified Human Gastrointestinal Proteome by DPCfam clustering

Contact Info

Product

Resources

About