The pitfalls of negative data bias for the T-cell epitope specificity challenge

Dens, Ceder; Laukens, Kris; Bittremieux, Wout; Meysman, Pieter

doi:10.1101/2023.04.06.535863

Cited by 8 publications

(9 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The negative data generation strategy is worth noting. The utilization of a background TCR pool might lead to bias and overestimation of model performance (Dens et al, 2023;Moris et al, 2021), so we used random shuffle strategy to obtain our negative pairs. This strategy might bring false negative samples although the possibility is relatively low.…”

Section: Discussionmentioning

confidence: 99%

TEPCAM: Prediction of T‐cell receptor–epitope binding specificity via interpretable deep learning

Chen,

Zhao,

Lin

et al. 2023

Protein Science

View full text Add to dashboard Cite

The recognition of T‐cell receptor (TCR) on the surface of T cell to specific epitope presented by the major histocompatibility complex is the key to trigger the immune response. Identifying the binding rules of TCR–epitope pair is crucial for developing immunotherapies, including neoantigen vaccine and drugs. Accurate prediction of TCR–epitope binding specificity via deep learning remains challenging, especially in test cases which are unseen in the training set. Here, we propose TEPCAM (TCR–EPitope identification based on Cross‐Attention and Multi‐channel convolution), a deep learning model that incorporates self‐attention, cross‐attention mechanism, and multi‐channel convolution to improve the generalizability and enhance the model interpretability. Experimental results demonstrate that our model outperformed several state‐of‐the‐art models on two challenging tasks including a strictly split dataset and an external dataset. Furthermore, the model can learn some interaction patterns between TCR and epitope by extracting the interpretable matrix from cross‐attention layer and mapping them to the three‐dimensional structures. The source code and data are freely available at https://github.com/Chenjw99/TEPCAM.

show abstract

Section: Discussionmentioning

confidence: 99%

TEPCAM: Prediction of T‐cell receptor–epitope binding specificity via interpretable deep learning

Chen,

Zhao,

Lin

et al. 2023

Protein Science

View full text Add to dashboard Cite

show abstract

“…Future progress in AIRR-ML research is required in areas as diverse as evaluating the impact of sample size on prediction accuracy (Pavlović et al 2021;Kanduri et al 2022), negative dataset definition (Montemurro, Jessen, and Nielsen 2022;Deng et al 2023;Dens et al 2023), and unbiased estimation of prediction accuracy (Meysman et al 2023;Moris et al 2021b). For all of these use cases, LIgO simulations may be employed for benchmarking and developing interpretable AIRR-ML methods.…”

Section: Discussionmentioning

confidence: 99%

“…For example, (i) if the train and test set overlap or contain highly similar sequences, then the accuracy of the trained ML model may be overly optimistic (a case of data leakage). (ii) If the training data is not comprehensive and representative, it may lead to generalization problems and underperformance on unseen data (Montemurro, Jessen, and Nielsen 2022;Deng et al 2023;Dens et al 2023;Moris et al 2021a;Robert et al 2022;Meysman et al 2023;Walsh, Pollastri, and Tosatto 2016;Petti and Eddy 2022;A. Weber, Born, and Rodriguez Martínez 2021).…”

Section: Comparison Of Simulation Strategies Implemented In the Ligo ...mentioning

confidence: 99%

“…Similarly to repertoire classification, it remains challenging to predict antigen binding for individual AIRs (receptor-based classification) given, for instance, the current scarcity of the antigen-labeled data, as well as challenges in quantification of generalization and negative data definition (Robert et al, n.d.;Akbar et al 2021;Dash et al 2017;Glanville et al 2017;A. Weber, Born, and Rodriguez Martínez 2021;Moris et al 2021a;Akbar, Bashour, et al 2022;Hudson et al 2023;Dens et al 2023).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Simulation of adaptive immune receptors and repertoires with complex immune information to guide the development and benchmarking of AIRR machine learning

Chernigovskaya,

Pavlović,

Kanduri

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Machine-learning methods (ML) have shown great potential in the adaptive immune receptor repertoire (AIRR) field. However, there is a lack of large-scale ground-truth experimental AIRR data suitable for AIRR-ML-based disease diagnostics and therapeutics discovery. Simulated ground-truth AIRR data are required to complement the development and benchmarking of robust and interpretable AIRR-ML approaches where experimental data is inaccessible or insufficient as of yet. The challenge for simulated data to be useful is the ability to incorporate key features observed in experimental repertoires. These features, such as complex antigen or disease-associated immune information, cause AIRR-ML problems to be challenging. Here, we introduce LIgO, a modular software suite, which simulates AIRR data for the development and benchmarking of AIRR-based machine learning. LIgO incorporates different types of immune information both on the receptor and the repertoire level and preserves native-like generation probability distribution. Additionally, LIgO assists users in determining the computational feasibility of their simulations. We show two examples where LIgO simulation supports the development and validation of AIRR-ML methods: (1) how individuals carrying out-of-distribution immune information impacts receptor-level prediction performance and (2) how immune information co-occurring in the same AIRs have an impact on the performance of conventional receptor-level encoding and repertoire-level classification approaches. The LIgO software guides the advancement and assessment of interpretable AIRR-ML methods.

show abstract

“…However, in the TCR-peptide recognition community, it is well known that the cross-reactivity of TCRs is a significant challenge in the TCR-peptide recognition problem and it can pose difficulties for many models in the pre-processing stage 2 . Experimental methods can have a high false-negative rate 1 , resulting in many potential cross-reactivity in existing known binding TCRs, making PU Learning more challenging. In the absence of experimental negatives, two main negative sampling strategies have emerged, including reshuffling based on positive pairs (first strategy) and randomly drawing from background repertories (second strategy).…”

mentioning

confidence: 99%

Reply to: The pitfalls of negative data bias for the T-cell epitope specificity challenge

Gao

Dong

et al. 2023

Preprint

View full text Add to dashboard Cite

We noticed that recently Pieter Meysman et al indicated the negative data sampling issue in T-cell epitope specificity prediction. In light of the limited data available in this area, the negative sampling issue is generally important for biological data modeling, since biological experimental tests intend to record the positive results while ignore the negative results. We appreciated their efforts to raise this point for T-cell epitope specificity modeling, which is known clearly by the community that different negative data sampling strategy will influence the prediction results. Therefore, proper negative data sampling strategy should be carefully selected, and this is exactly what PanPep has noticed, emphasized and performed. In short, as for the two commonly used negative sampling strategy, i.e., reshuffling based on positive pairs (first strategy) and randomly drawing from background repertories (second strategy), PanPep prefers to select the second strategy, and the rational has been clearly indicated in the manuscript. Now we would like to clarify this point further by formulating this problem as a PU learning and calling for more attentions on this point.

show abstract

The pitfalls of negative data bias for the T-cell epitope specificity challenge

Cited by 8 publications

References 16 publications

TEPCAM: Prediction of T‐cell receptor–epitope binding specificity via interpretable deep learning

TEPCAM: Prediction of T‐cell receptor–epitope binding specificity via interpretable deep learning

Simulation of adaptive immune receptors and repertoires with complex immune information to guide the development and benchmarking of AIRR machine learning

Reply to: The pitfalls of negative data bias for the T-cell epitope specificity challenge

Contact Info

Product

Resources

About