External validation of WGS-based antimicrobial susceptibility prediction tools, KOVER-AMR and ResFinder 4.1, for Escherichia coli clinical isolates

Verschuuren, Tess D.; Bosch, Thijs; Mascaro, V.; Willems, Rob J. L.; Kluytmans, Jan

doi:10.1016/j.cmi.2022.05.024

Cited by 15 publications

(16 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Even though AMR gene finder tools are designed to identify the presence of AMR genes in genomic data, their results are frequently used to directly infer AMR phenotype in literature [40, 6, 17, 43]. We examined the accuracy of predicting AMR phenotype solely based on the presence/absence of AMR genes for 23 antibiotics and 16,950 genomes, from organisms with laboratory-derived MIC data.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Accessory genes define species-specific pathways to antibiotic resistance

Dillon

Dimonaco

Creevey

2023

Preprint

View full text Add to dashboard Cite

Background: The rise of antimicrobial resistance (AMR) is a growing concern globally and a deeper understanding of AMR gene carriage vs usage is vital for future responses to reduce the spread of AMR. Identification of AMR phenotype by laboratory-based assays are often hindered by difficulties in establishing cultures. This issue could be resolved by rapid computational assessment of an organism's genome, however, AMR gene finder tools are not intended to infer AMR phenotype which is likely to be a product of multiple gene interactions. Methods: To understand the importance of multi-gene interactions to the relationship between AMR genotype and AMR phenotype, we applied machine learning approaches to 16,950 genomes from microbial isolates representing 28 different genera with 1.2 million corresponding laboratory-determined MICs for 23 different antibiotics. We then elucidated the genomic paths to phenotypic antimicrobial resistance with the aim of allowing for the development of rapid determination of AMR phenotype from genomes or even whole microbiomes. Results: The application of machine learning models resulted in a >1.5-fold increase in average prediction accuracy of AMR phenotype across the 23 antibiotic models. Interpretation of these models revealed 528 distinct genomic pathways to phenotypic resistance, many of which were species-specific and involved genes which have not previously been associated with AMR phenotype. This is the first study to demonstrate the utility of machine learning models in the prediction of AMR phenotype for a wide range of clinically relevant organisms and antibiotics. This could be applied as a rapid and affordable alternative to culture-based techniques, estimating taxonomy in addition to AMR phenotype, and providing real-time monitoring of multi-drug resistant pathogens.

show abstract

Section: Resultsmentioning

confidence: 99%

“…The average precision was 56.2% and the average recall was 61.2%, which highlights key flaws in using the tool to predict AMR phenotype. While RGI is not designed to identify AMR phenotype but rather the AMR genotype, its results are often inferred as phenotypic resistance for genomes [40, 6, 17, 43] and metagenomes [22, 47].…”

Section: Discussionmentioning

confidence: 99%

Accessory genes define species-specific pathways to antibiotic resistance

Dillon

Dimonaco

Creevey

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…None of the antibiotic classes we evaluated met the FDA criteria for acceptable maj and vmj discrepancy rates. In addition to the possible improvements described above, more granular drug-level classification of ARGs should be a priority for the AMRFinder tool and would likely improve predictive performance (highlighted by the overall slightly better performance of ResFinder in a recent validation study [10] and by better performance of existing tools when using curated gene-drug associations [9,28]). Our study also highlights potential phenotypic differences for alleles of AMR-associated gene families which differ by synonymous mutations, suggesting that classification of ARGs using amino-acid sequences alone should be avoided.…”

Section: Discussionmentioning

confidence: 99%

“…Several studies have investigated the performance of such databases to predict phenotype from genotype for E. coli [8, 9], demonstrating the associated challenges. Verschuuren et al recently demonstrated the inability of the Resfinder tool to meet FDA specifications for very major/major error rates for most antibiotics (n=234 isolates, selected for resistance to third-generation cephalosporins[10]). This work highlighted particularly poor performance predicting AMR phenotype for beta lactam beta-lactamase inhibitor (BL-BLI) combination drugs, replicating a finding from earlier studies[11, 12].…”

Section: Introductionmentioning

confidence: 99%

Exploring the extent of uncatalogued genetic variation in antimicrobial resistance gene families inEscherichia coli

Lipworth

Crook

Walker

et al. 2023

Preprint

View full text Add to dashboard Cite

Background Antimicrobial resistance (AMR) in E. coli is a global problem associated with substantial morbidity and mortality. AMR-associated genes are typically annotated based on similarity to a variants in a curated reference database with an implicit assumption that uncatalogued genetic variation within these is phenotypically unimportant. In this study we evaluated the potential for discovering new AMR-associated gene families and characterising variation within existing ones to improve genotype-to-susceptibility-phenotype prediction in E. coli. Methods We assembled a global dataset of 9001 clinical E. coli sequences of which 8586 had linked antibiotic susceptibility data. Raw reads were assembled using Shovill and AMR genes extracted using the NCBI AMRFinder tool. Mash was used to calculate the similarity between extracted genes using Jaccard distances. We empirically reclustered extracted gene sequences into AMR-associated gene families (≥70% match) and alleles (ARGs, 100% match). Results The performance of the AMRFinder database for genotype-to-phenotype predictions using strict 100% identity and coverage thresholds did not meet FDA thresholds for any of the eight antibiotics evaluated. Relaxing filters to default settings improved sensitivity with a specificity cost. For all antibiotics, a small number of genes explained most resistance although a proportion could not be explained by known ARGs; this ranged from 75.1% for co-amoxiclav to 3.4% for ciprofloxacin. Only 17,177/36,637 (47%) of ARGs detected had a 100% identity and coverage match in the AMRFinder database. After empirically reclassifying genes at 100% nucleotide sequence identity, we identified 1292 unique ARGs of which 158 (12%) were present ≥10 times, 374 (29%) were present 2-9 times and 760 (59%) only once. Simulated accumulation curves revealed that discovery of new (100%-match) ARGs present more than once in the dataset plateaued relatively quickly whereas new singleton ARGs were discovered even after many thousands of isolates had been included. We identified a strong correlation (Spearman coefficient 0.76 (95% CI 0.72-0.79, p<0.001)) between the number of times an ARG was observed in Oxfordshire and the number of times it was seen internationally, with ARGs that were observed ≥7 times in Oxfordshire always being found elsewhere. Finally, using the example of blaTEM-1, we demonstrated that uncatalogued variation, including synonymous variation, is associated with potentially important phenotypic differences (e.g. two common, uncatalogued blaTEM-1 alleles with only synonymous mutations compared to the known reference were associated with reduced resistance to co-amoxiclav [aOR 0.57, 95%CI 0.34-0.93, p=0.03] and piperacillin-tazobactam [aOR 0.54, 95%CI 0.32-0.87, p=0.01]). Conclusions Overall we highlight substantial uncatalogued genetic variation with respect to known ARGs, although a relatively small proportion of these alleles are repeatedly observed in a large international dataset suggesting strong selection pressures. The current approach of using fuzzy matching for ARG detection, ignoring the unknown effects of uncatalogued variation, is unlikely to be acceptable for future clinical deployment. The association of synonymous mutations with potentially important phenotypic differences suggests that relying solely on amino acid-based gene detection to predict resistance is unlikely to be sufficient. Finally, the inability to explain all resistance using existing knowledge highlights the importance of new target gene discovery.

show abstract

“…Accurate WGS-based profiling of complete AMR gene content and prediction of susceptibility phenotypes would represent an attractive option for other commonly encountered clinical bacterial pathogens, such as Enterobacterales, including Escherichia coli . However, many of these pathogens, together with the antimicrobials commonly used to treat them, have proved more challenging, with methods designed for this yet to meet the standards required to be used in clinical practice [4–6].…”

Section: Introductionmentioning

confidence: 99%

Discordance between different bioinformatic methods for identifying resistance genes from short-read genomic data, with a focus on Escherichia coli

Davies,

Swann,

Sheppard

et al. 2023

Microbial Genomics

View full text Add to dashboard Cite

Several bioinformatics genotyping algorithms are now commonly used to characterize antimicrobial resistance (AMR) gene profiles in whole-genome sequencing (WGS) data, with a view to understanding AMR epidemiology and developing resistance prediction workflows using WGS in clinical settings. Accurately evaluating AMR in Enterobacterales, particularly Escherichia coli , is of major importance, because this is a common pathogen. However, robust comparisons of different genotyping approaches on relevant simulated and large real-life WGS datasets are lacking. Here, we used both simulated datasets and a large set of real E. coli WGS data (n=1818 isolates) to systematically investigate genotyping methods in greater detail. Simulated constructs and real sequences were processed using four different bioinformatic programs (ABRicate, ARIBA, KmerResistance and SRST2, run with the ResFinder database) and their outputs compared. For simulation tests where 3079 AMR gene variants were inserted into random sequence constructs, KmerResistance was correct for 3076 (99.9 %) simulations, ABRicate for 3054 (99.2 %), ARIBA for 2783 (90.4 %) and SRST2 for 2108 (68.5 %). For simulation tests where two closely related gene variants were inserted into random sequence constructs, KmerResistance identified the correct alleles in 35 338/46 318 (76.3 %) simulations, ABRicate identified them in 11 842/46 318 (25.6 %) simulations, ARIBA identified them in 1679/46 318 (3.6 %) simulations and SRST2 identified them in 2000/46 318 (4.3 %) simulations. In real data, across all methods, 1392/1818 (76 %) isolates had discrepant allele calls for at least 1 gene. In addition to highlighting areas for improvement in challenging scenarios, (e.g. identification of AMR genes at <10× coverage, identifying multiple closely related AMR genes present in the same sample), our evaluations identified some more systematic errors that could be readily soluble, such as repeated misclassification (i.e. naming) of genes as shorter variants of the same gene present within the reference resistance gene database. Such naming errors accounted for at least 2530/4321 (59 %) of the discrepancies seen in real data. Moreover, many of the remaining discrepancies were likely ‘artefactual’, with reporting of cut-off differences accounting for at least 1430/4321 (33 %) discrepants. Whilst we found that comparing outputs generated by running multiple algorithms on the same dataset could identify and resolve these algorithmic artefacts, the results of our evaluations emphasize the need for developing new and more robust genotyping algorithms to further improve accuracy and performance.

show abstract

External validation of WGS-based antimicrobial susceptibility prediction tools, KOVER-AMR and ResFinder 4.1, for Escherichia coli clinical isolates

Cited by 15 publications

References 24 publications

Accessory genes define species-specific pathways to antibiotic resistance

Accessory genes define species-specific pathways to antibiotic resistance

Exploring the extent of uncatalogued genetic variation in antimicrobial resistance gene families inEscherichia coli

Discordance between different bioinformatic methods for identifying resistance genes from short-read genomic data, with a focus on Escherichia coli

Contact Info

Product

Resources

About