Organism-specific training improves performance of linear B-cell epitope prediction

Abstract

Motivation In silico identification of linear B-cell epitopes represents an important step in the development of diagnostic tests and vaccine candidates, by providing potential high-probability targets for experimental investigation. Current predictive tools were developed under a generalist approach, training models with heterogeneous data sets to develop predictors that can be deployed for a wide variety of pathogens. However, continuous advances in processing power and the increasing amount of epitope data for a broad range of pathogens indicate that training organism or taxon-specific models may become a feasible alternative, with unexplored potential gains in predictive performance. Results This paper shows how organism-specific training of epitope prediction models can yield substantial performance gains across several quality metrics when compared to models trained with heterogeneous and hybrid data, and with a variety of widely-used predictors from the literature. These results suggest a promising alternative for the development of custom-tailored predictive models with high predictive power, which can be easily implemented and deployed for the investigation of specific pathogens. Availability The data underlying this article, as well as the full reproducibility scripts, are available at https://github.com/fcampelo/OrgSpec-paper. The R package that implements the organism-specific pipeline functions is available at https://github.com/fcampelo/epitopes. Supplementary information Supplementary materials are available at Bioinformatics online.

Estimated limits of organism-specific training for epitope prediction

“…The results also show that organism-specific training outperforms generalist training (predictors from the literature, trained on peptides from a wide variety of pathogens) even when very small organism-specific data sets are available. The only systematic exception was the high observed performance of LBtope for the Hepatitis C Virus; However, as mentioned earlier, "part of the hold-out examples used to asses the performance of the models is present in the training data of LBtope (9.59% of the Hep C hold-out sequences are present in the LBtope training data set)" [45], which in the case of our experiments would result in some level of data leakage [49]. In addition to showing that organism-specific training outperforms heterogeneous and hybrid training, this work shows that adding unrelated data to organism-specific training sets decreases the generalisation performance of the resulting model when tasked with predicting epitopes for the target pathogen.…”

“…Epitope prediction models were developed by training Random Forest (RF) predictors on each of the training data sets outlined above, using Scikit-learn version 0.24.1 [48] under standard hyper-parameter values. The choice of Random Forest was based on preliminary experimentation, as documented in [45], and also to make this work more directly comparable with the results reported in that earlier one. The trained models were then used to generate predictions for the organism-specific hold-out data sets and prediction performance was assessed using multiple different performance measures, namely: Balanced Accuracy (BAL.ACC), Matthew's Correlation Coefficient (MCC), Area Under the Curve (AUC), Positive Predictive Value (PPV), Negative Predictive Value (NPV) and Sensitivity (SENS).…”

“…Data from three pathogens was taken from [45], specific to the organisms: Onchocerca volvulus (taxonomy ID: 6282; ), Epstein-Barr Virus (taxonomy ID: 10376) and Hepatitis C Virus (taxonomy ID: 11102). These data sets were generated based on the full XML export of the IEDB retrieved on the 10th of October 2020, and filtered according to the criteria listed in [45] (section 2.1, "Data sets").…”

“…Data from three pathogens was taken from [45], specific to the organisms: Onchocerca volvulus (taxonomy ID: 6282; ), Epstein-Barr Virus (taxonomy ID: 10376) and Hepatitis C Virus (taxonomy ID: 11102). These data sets were generated based on the full XML export of the IEDB retrieved on the 10th of October 2020, and filtered according to the criteria listed in [45] (section 2.1, "Data sets"). The available data was split at the protein level, with entries coming from the same protein, or from proteins exhibiting sequence coverage and similarity greater than 80%, always placed in the same split.…”

“…Currently, most epitope prediction models are trained on large heterogeneous data sets made up of observations from multiple organisms including: prokaryotes, viruses, fungi, protozoan, humans and other eukaryotes. However, in a recent work [45] we have shown that training models on smaller organism-specific data sets may improve prediction performance. In that work, organism-specific models were developed for three different organisms: the nematode Onchocerca volvulus, the Epstein-Barr Virus and the Hepatitis C Virus.…”