Benchmarking Variant Identification Tools for Plant Diversity Discovery

Abstract

Background The ability to accurately and comprehensively identify genomic variations is critical for plant studies utilizing high-throughput sequencing. Most bioinformatics tools for processing next-generation sequencing data were originally developed and tested in human studies, raising questions as to their efficacy for plant research. A detailed evaluation of the entire variant calling pipeline, including alignment, variant calling, variant filtering, and imputation was performed on different programs using both simulated and real plant genomic datasets. Results A comparison of SOAP2, Bowtie2, and BWA-MEM found that BWA-MEM was consistently able to align the most reads with high accuracy, whereas Bowtie2 had the highest overall accuracy. Comparative results of GATK HaplotypCaller versus SAMtools mpileup indicated that the choice of variant caller affected precision and recall differentially depending on the levels of diversity, sequence coverage and genome complexity. A cross-reference experiment of S. lycopersicum and S. pennellii reference genomes revealed the inadequacy of single reference genome for variant discovery that includes distantly-related plant individuals. Machine-learning-based variant filtering strategy outperformed the traditional hard-cutoff strategy resulting in higher number of true positive variants and fewer false positive variants. The 2-step imputation which utilized a set of high-confidence SNPs as the reference panel showed up to 60% higher accuracy than direct LD-based imputation method. Conclusions Programs in the variant discovery pipeline have different performance on plant genomic dataset. Choice of the programs is subjected to the goal of the study and available resources. This study serves as an important guiding information for plant biologists utilizing next-generation sequencing data for diversity characterization and crop improvement.

Comparison of read mapping and variant calling tools for the analysis of plant NGS data

“…In terms of sensitivity, GATK in combination with CLC-mapper, Novoalign and BWA-MEM yielded the best and most consistent results across all evaluated data sets. These results are in line with a recent study showing that GATK often outperformed SAMtools in terms of sensitivity, precision, and called raw InDels 44 . A high sensitivity is essential when a high number of true positives variations accelerates the power of the analyses, e.g.…”

“…Hence, we performed a benchmarking study to provide comparable data showing the performance of combinations of frequently applied mappers and variant callers (variant calling pipelines) on plant data sets. A previous report 44 is extended by providing data about the performance of additional tools both for the mapping and variant calling step.…”

“…Due to substantial differences in the nucleotide composition, a dedicated benchmarking on plant genomes sequences is advised. A recent study compared the performance of BWA-MEM 22 , SOAP2 24 , and Bowtie2 21 with the two variant callers GATK [29][30][31][32] and SAMtools/BCFtools 27 on simulated and real tomato datasets 44 . To expand the sparse knowledge about the performance of other read mapping and variant calling tools on plant data, we set out to perform a systematic comparison.…”

Reanalysis of genome sequences of tomato accessions and its wild relatives: development of Tomato Genomic Variation (TGV) database integrating SNPs and INDELs polymorphisms

“…They, too showed that BWA-MEM outperformed other aligners in mapping percentage and accuracy. Alike, for variant calling Wu et al (2019) also observed the increase in the number of SNPs when using GATK HaplotypeCaller in comparison to the SAMtools. They also showed that variant calling and filtering using GATK HaplotypeCaller outperformed SAMtools in terms of precision and recalling of variants in highly diverse samples.…”

“…Recently Wu et al (2019) compared three sequence aligners, BWA-MEM, Bowtie2 and SOAP2, and two variant callers GATK HaplotypeCaller and SAMtools mpileup for benchmarking tools for plant diversity discovery using domesticated, wild-relatives, and simulated genomic datasets of tomato. They, too showed that BWA-MEM outperformed other aligners in mapping percentage and accuracy.…”