DECA: scalable XHMM exome copy-number variant calling with ADAM and Apache Spark

Abstract

Background XHMM is a widely used tool for copy-number variant (CNV) discovery from whole exome sequencing data but can require hours to days to run for large cohorts. A more scalable implementation would reduce the need for specialized computational resources and enable increased exploration of the configuration parameter space to obtain the best possible results. Results DECA is a horizontally scalable implementation of the XHMM algorithm using the ADAM framework and Apache Spark that incorporates novel algorithmic optimizations to eliminate unneeded computation. DECA parallelizes XHMM on both multi-core shared memory computers and large shared-nothing Spark clusters. We performed CNV discovery from the read-depth matrix in 2535 exomes in 9.3 min on a 16-core workstation (35.3× speedup vs. XHMM), 12.7 min using 10 executor cores on a Spark cluster (18.8× speedup vs. XHMM), and 9.8 min using 32 executor cores on Amazon AWS’ Elastic MapReduce. We performed CNV discovery from the original BAM files in 292 min using 640 executor cores on a Spark cluster. Conclusions We describe DECA’s performance, our algorithmic and implementation enhancements to XHMM to obtain that performance, and our lessons learned porting a complex genome analysis application to ADAM and Spark. ADAM and Apache Spark are a performant and productive platform for implementing large-scale genome analyses, but efficiently utilizing large clusters can require algorithmic optimizations and careful attention to Spark’s configuration parameters.

Framing Apache Spark in life sciences

“…The table shows that apart from some tools that reports tests only on a multi-core workstation ( [16] , [17] , [18] , [19] ), Spark has been widely used to implement tools aimed at parallelizing the computation on a distributed computing environment. Most of these tools have been specifically devised for, or tested on, a cloud environment ( [20] , [21] , [22] , [23] , [24] , [25] , [26] , [27] , [28] [29] , [30] , [31] , [32] [33] , [34] , [35] , [36] , [37] ). Being the increasing availability of IaaS (Infrastructure as a Service) cloud computing services, it is desirable that the released tools are commonly designed to be supported also by such infrastructures.…”

“…CMAN Ext. tools/frameworks Genomics genome assembly SORA [20] de novo genome assembly GraphX ✓ - ✓ - - - variant calling DECA [21] copy number variantion discovery MLlib ✓ - ✓ - - ADAM ADS-HCSpark [48] SNPs and indels calling - ✓ - - - - - SparkGA2 [22] variant calling - ✓ - ✓ ✓ - - SparkRA [49] GATK best-practices pipeline - ✓ - - - - - DeepVariant on Spark [23] SNPs and indels calling - ✓ ✓ ✓ ✓ - Apache Parquet VC@Scale [24] SNPs and indels calling - ✓ ✓ ✓ - - Apache Arrow Halvade Somatic [25] somatic variant calling - ✓ - ✓ - - - …”

“…Linderman et al (2019) [21] propose DECA, a horizontally scalable implementation of the XHMM algorithm –which is aimed at discovering copy-number variants (CNVs) of the exome. DECA is based on Spark and ADAM [47] , the latter being an in-memory distributed computing framework for genome analysis built on Spark.…”

Genomically Aided Diagnosis of Severe Developmental Disorders

“…Rather unexpectedly, it has proven possible to use trio WES to identify de novo and inherited copy number variants (4,34,75). Intragenic or genic deletions or duplications account for approximately 2% of causative alleles in developmental disorders (37,57), but this figure may be under-ascertained due to technical issues related to short-read sequencing.…”