Benchmarking solutions to the T-cell receptor epitope prediction problem: IMMREP22 workshop report

Meysman, Pieter; Barton, Justin; Bravi, Barbara; Cohen-Lavi, Liel; Карнаухов, В. К.; Lilleskov, Elias; Montemurro, Alessandro; Nielsen, Morten; Mora, Thierry; Pereira, Paul; Postovskaya, Anna; Martínez, María Rodríguez; Fernández-de-Cossio-Díaz, Jorge; Vujkovic, Alexandra; Walczak, Aleksandra M.; Weber, Anna; Yin, Rose; Eugster, Anne; Sharma, Virag

doi:10.1101/2022.10.27.514020

Cited by 9 publications

(16 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We showed that this model achieved comparable performance to the one of NetTCR, while being very simple. These results align with previous findings ( 17 , 26 , 29 , 30 ). A closer analysis of our results revealed that TCRbase performed at par with NetTCR when separating positive versus 10X negative TCRs; however, the gap in performance between the two models was enlarged on the positives versus swapped negatives prediction task, where NetTCR significantly outperformed TCRbase.…”

Section: Discussionsupporting

confidence: 93%

NetTCR-2.1: Lessons and guidance on how to develop models for TCR specificity predictions

Montemurro

Jessen²,

Nielsen³

2022

Front. Immunol.

Self Cite

View full text Add to dashboard Cite

T cell receptors (TCR) define the specificity of T cells and are responsible for their interaction with peptide antigen targets presented in complex with major histocompatibility complex (MHC) molecules. Understanding the rules underlying this interaction hence forms the foundation for our understanding of basic adaptive immunology. Over the last decade, efforts have been dedicated to developing assays for high throughput identification of peptide-specific TCRs. Based on such data, several computational methods have been proposed for predicting the TCR-pMHC interaction. The general conclusion from these studies is that the prediction of TCR interactions with MHC-peptide complexes remains highly challenging. Several reasons form the basis for this including scarcity and quality of data, and ill-defined modeling objectives imposed by the high redundancy of the available data. In this work, we propose a framework for dealing with this redundancy, allowing us to address essential questions related to the modeling of TCR specificity including the use of peptide- versus pan-specific models, how to best define negative data, and the performance impact of integrating of CDR1 and 2 loops. Further, we illustrate how and why it is strongly recommended to include simple similarity-based modeling approaches when validating an improved predictive power of machine learning models, and that such validation should include a performance evaluation as a function of “distance” to the training data, to quantify the potential for generalization of the proposed model. The conclusion of the work is that, given current data, TCR specificity is best modeled using peptide-specific approaches, integrating information from all 6 CDR loops, and with negative data constructed from a combination of true and mislabeled negatives. Comparing such machine learning models to similarity-based approaches demonstrated an increased performance gain of the former as the “distance” to the training data was increased; thus demonstrating an improved generalization ability of the machine learning-based approaches. We believe these results demonstrate that the outlined modeling framework and proposed evaluation strategy form a solid basis for investigating the modeling of TCR specificities and that adhering to such a framework will allow for faster progress within the field. The final devolved model, NetTCR-2.1, is available at https://services.healthtech.dtu.dk/service.php?NetTCR-2.1.

show abstract

Section: Discussionsupporting

confidence: 93%

NetTCR-2.1: Lessons and guidance on how to develop models for TCR specificity predictions

Montemurro

Jessen²,

Nielsen³

2022

Front. Immunol.

Self Cite

View full text Add to dashboard Cite

show abstract

“…prediction whether random TCRs bind to a specific peptide). Meysman et al have compared superficially different approaches to TCR-pMHC binding (17), but also raised the importance of a truly independent benchmark. They reveal that additional information like CDR1/2 improved the prediction, but they did not investigate the role that imbalance, size or overtraining might have on model performance by using those additional features within the used dataset.…”

Section: Discussionmentioning

confidence: 99%

Performance comparison of TCR-pMHC prediction tools reveals a strong data dependency

Deng

Ly²,

Abdollahi³

et al. 2022

Preprint

View full text Add to dashboard Cite

The interaction of T-cell receptors with peptide-major histocompatibility complex molecules plays a crucial role in adaptive immune responses. Currently there are various models aiming at predicting TCR-pMHC binding, while a standard dataset and procedure to compare the performance of these approaches is still missing. In this work we provide a general method for data collection, preprocessing, splitting and generation of negative examples, as well as comprehensive datasets to compare TCR-pMHC prediction models. We collected, harmonized, and merged all the major publicly available TCR-pMHC binding data and compared the performance of five state-of-the-art deep learning models (TITAN, NetTCR, ERGO, DLpTCR and ImRex) using this data. Our performance evaluation focuses on two scenarios: 1) different splitting methods for generating training and testing data to assess model generalization and 2) different data versions that vary in size and peptide imbalance to assess model robustness. Our results indicate that the five contemporary models do not generalize to peptides that have not been in the training set. We can also show that model performance is strongly dependent on the data balance and size, which indicates a relatively low model robustness. These results suggest that TCR-pMHC binding prediction remains highly challenging and requires further high quality data and novel algorithmic approaches.

show abstract

“…In this case, one uses CDR3 α sequences from healthy TCR repertoires and samples of antigen-specific CDR3 α as respectively the background and the selected dataset. DiffRBM models for the α chain reach a discrimination performance comparable to the one for the β chain, as shown in [Meysman et al, 2022]. On the same footing, with single-cell TCR sequencing data becoming increasingly available, our approach could also be extended to modeling the pairs of TCR α and β chains, which have been suggested to play a synergistic role in determining antigen specificity [Carter et al, 2019, Montemurro et al, 2021, Milighetti et al, 2021].…”

Section: Discussionmentioning

confidence: 92%

“…An approach like diffRBM (similarly to SONIA) returns peptide-specific models and hence needs to be trained on TCRs known to recognize a certain peptide for any prediction of specificity to that peptide. Furthermore, the performance by diffRBM decreases compared to other methods when we look at a different task, the one of discriminating a set of antigen-specific receptors from receptors with another antigen specificity [Meysman et al, 2022]. This is due to the fact that our approach, in contrast to other methods [Gielis et al, 2019, Montemurro et al, 2021, Weber et al, 2021], is unsupervised, meaning that it is not trained using receptors of different antigen specificity as negatives against which the positives should be discriminated.…”

Section: Resultsmentioning

confidence: 97%

“…It is at this task that diffRBM performs as well as the state-of-the-art methods. The performance by diffRBM decreases when we consider as negatives receptors with a different antigen specificity [Meysman et al, 2022], because the differential units can learn features shared between antigen-specific sequences ( e.g ., being expressed by CD8+ T cells). More fine-tuned choices of the background ( e.g ., with only CD8+ T-cell receptors) would be needed to improve performance for this particular task.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Learning the differences: a transfer-learning approach to predict antigen immunogenicity and T-cell receptor specificity

Bravi

Gioacchino

Fernández-de-Cossio-Díaz

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Antigen immunogenicity and the specificity of binding of T-cell receptors to antigens are key properties underlying effective immune responses. Here we propose diffRBM, an approach based on transfer learning and Restricted Boltzmann Machines, to build sequence-based predictive models of these properties. DiffRBM is designed to learn the distinctive patterns in amino acid composition that, one the one hand, underlie the antigen's probability of triggering a response, and on the other hand the T-cell receptor's ability to bind to a given antigen. We show that the patterns learnt by diffRBM allow us to predict putative contact sites of the antigen-receptor complex. We also discriminate immunogenic and non-immunogenic antigens, antigen-specific and generic receptors, reaching performances that compare favorably to existing sequence-based predictors of antigen immunogenicity and T-cell receptor specificity. More broadly, diffRBM provides a general framework to detect, interpret and leverage selected features in biological data.

show abstract

Benchmarking solutions to the T-cell receptor epitope prediction problem: IMMREP22 workshop report

Cited by 9 publications

References 21 publications

NetTCR-2.1: Lessons and guidance on how to develop models for TCR specificity predictions

NetTCR-2.1: Lessons and guidance on how to develop models for TCR specificity predictions

Performance comparison of TCR-pMHC prediction tools reveals a strong data dependency

Learning the differences: a transfer-learning approach to predict antigen immunogenicity and T-cell receptor specificity

Contact Info

Product

Resources

About