Learning From Limited Data: Towards Best Practice Techniques for Antimicrobial Resistance Prediction From Whole Genome Sequencing Data

Abstract

Antimicrobial resistance prediction from whole genome sequencing data (WGS) is an emerging application of machine learning, promising to improve antimicrobial resistance surveillance and outbreak monitoring. Despite significant reductions in sequencing cost, the availability and sampling diversity of WGS data with matched antimicrobial susceptibility testing (AST) profiles required for training of WGS-AST prediction models remains limited. Best practice machine learning techniques are required to ensure trained models generalize to independent data for optimal predictive performance. Limited data restricts the choice of machine learning training and evaluation methods and can result in overestimation of model performance. We demonstrate that the widely used random k-fold cross-validation method is ill-suited for application to small bacterial genomics datasets and offer an alternative cross-validation method based on genomic distance. We benchmarked three machine learning architectures previously applied to the WGS-AST problem on a set of 8,704 genome assemblies from five clinically relevant pathogens across 77 species-compound combinations collated from public databases. We show that individual models can be effectively ensembled to improve model performance. By combining models via stacked generalization with cross-validation, a model ensembling technique suitable for small datasets, we improved average sensitivity and specificity of individual models by 1.77% and 3.20%, respectively. Furthermore, stacked models exhibited improved robustness and were thus less prone to outlier performance drops than individual component models. In this study, we highlight best practice techniques for antimicrobial resistance prediction from WGS data and introduce the combination of genome distance aware cross-validation and stacked generalization for robust and accurate WGS-AST.

The Relative Power of Structural Genomic Variation versus SNPs in Explaining the Quantitative Trait Growth in the Marine Teleost Chrysophrys auratus

“…This problem is described by the fact that it is generally more cost and time effective to screen for a large number of variants within an individual than it is to screen large numbers of individuals [ 62 ] as is common in the fields of, e.g., neuroimaging, genomics, motion tracking, eye tracking, and many other technology-based data collection methods that have led to a torrent of high-dimensional datasets. This is a well-known area where classical machine learning algorithms do not perform well [ 63 , 64 ]. However, despite small sample sizes being common and the fact that limited data are problematic for pattern recognition, only a limited number of papers have systematically investigated how the machine learning validation process should be designed to help avoid optimistic performance estimates.…”

Genome-Wide Mutation Scoring for Machine-Learning-Based Antimicrobial Resistance Prediction

“…Organism–compound datasets with fewer than 100 susceptible and 100 resistant isolates were excluded. Filtered datasets were partitioned into training and test sets (80%:20%) using a genome-distance-based method [ 17 ]. This dataset partitioning method is designed to reduce similarity between the training and the test dataset.…”

“…ML-based WGS-AST typically uses nucleotide k-mer representations of either input genome assemblies or raw sequencing reads [ 14 , 15 , 16 , 17 , 18 ]. K-mer sets have been successfully used for various bioinformatics analyses, ranging from species identification [ 19 ] to genome assembly [ 20 ], as they offer advantages in computing efficiency and speed.…”

“…While the addition of training data is the most straightforward strategy, it is not always feasible. Previously, we explored the utility of different ML algorithms and their possible combinations [ 17 ]. In this study, we investigated improvements of WGS-AST ML models via the feature-space engineering approach.…”

Predictive Antibiotic Susceptibility Testing by Next-Generation Sequencing for Periprosthetic Joint Infections: Potential and Limitations

“…Only a subset of ML algorithms is capable of effectively making use of high-dimensional data while minimizing overfitting [ 30 ]. Likewise, rigorous validation on independently sampled datasets is required for robust estimation of model performance in the general case [ 45 , 71 ]. While the increasing availability of datasets with both NGS and AST data will help in improving performance and generalizability, more research is required to establish guidelines for sampling and validation of pAST ML models that can support clinical applications.…”