Evaluating performance of existing computational models in predicting CD8+ T cell pathogenic epitopes and cancer neoantigens

Abstract

T cell recognition of a cognate peptide-MHC complex (pMHC) presented on the surface of infected or malignant cells are of utmost importance for mediating robust and long-term immune responses. Accurate predictions of cognate pMHC targets for T Cell Receptors (TCR) would greatly facilitate identification of vaccine targets for both pathogenic diseases as well as personalized cancer immunotherapies. Predicting immunogenic epitopes therefore has been at the centre of intensive research for the past decades but has proven challenging. Although numerous models have been proposed, performance of these models has not been systematically evaluated and their success rate in predicting epitopes for humans has not been measured and compared.In this study, we curate and present a ‘standard-dataset’ of class I MHC epitopes for evaluating the performance of immunogenicity classifiers. Using this data, we present a systematic evaluation of performance of several publicly available models which are commonly used to predict CD8+ T cell epitopes in the context of both pathogens and cancers. The benchmarking is performed in two different settings: a pan-HLA setting in which all models are trained and evaluated by a pool of peptides from multiple HLA types, and a HLA-specific setting in which models are trained and evaluated by peptides from HLA-A02:01. In the pan-HLA setting, we observe that the best performing model achieves 75.6% accuracy suggesting considerable room for improvement. In the HLA-specific setting we observe a substantial reduction in performance with a mean of −11.9%. Our work shows that existing models exhibit suboptimal performance for predicting immunogenic cancer neoantigens. We then further interrogate the underlying problems of model predictions and highlight HLA-bias as a main source of variation amongst other issues associated with the design of models and/or to their training data.

De-risking clinical trial failure through mechanistic simulation

“…Off-rates predicted by online utilities were inconsistent not only with measured values but with each other, both in order of magnitude and in rank order of peptides, though they correctly predicted which peptides would bind poorly. Previous authors have made similar observations; a recent study evaluating performance of computational models in predicting CD8+ epitopes found that no existing algorithm performs ‘substantially’ better than random [ 22 ], and a study introducing MHCflurry, an MHC-I binding affinity prediction package that the authors present as an improvement over NetMHC and NetMHCpan, found that all three algorithms predict affinities several orders of magnitude away from measured values for most simulated peptides [ 23 ]. This shows that care must be taken when extrapolating the results of machine learning outside of the dataset used to train such models.…”

A robust deep learning platform to predict CD8+ T-cell epitopes

“…We previously reported that the existing immunogenicity algorithms showed suboptimal performance in predicting epitopes from both cancer and an emerging viral pathogen 7 . We then attributed the poor performance to divergent discriminative features between cancer neoepitopes and pathogenic epitopes in directionality or magnitudes 7,23 .…”

A systems approach evaluating the impact of SARS-CoV-2 variant of concern mutations on CD8+ T cell responses

“…While in silico models that predict antigen presentation can be highly accurate (such as netMHCpan), state-of-the-art models predicting the subset of HLA ligands that then invoke T cell responses possess limited accuracy[36]. We recently developed TRAP, a Convolutional Neural Network (CNN) model that offers improved predictions of T cell recognition potential of HLA-I presented 9- and 10-mer peptides.…”

“…In that light, in silico studies e.g., the work by Nersisyan et al [29] compared all theoretical HLA ligands across specific VOC and concluded that T cell responses to Omicron were likely to be maintained effectively. However, not all HLA ligands can invoke T cell responses [35,36]. Furthermore, studies such as those of Naranbhai et al [17] and Reynolds et al [18] observed considerable numbers of patients with impaired T cell responses to Omicron infection [52].…”