A Primer on High-Throughput Computing for Genomic Selection

Abstract

High-throughput computing (HTC) uses computer clusters to solve advanced computational problems, with the goal of accomplishing high-throughput over relatively long periods of time. In genomic selection, for example, a set of markers covering the entire genome is used to train a model based on known data, and the resulting model is used to predict the genetic merit of selection candidates. Sophisticated models are very computationally demanding and, with several traits to be evaluated sequentially, computing time is long, and output is low. In this paper, we present scenarios and basic principles of how HTC can be used in genomic selection, implemented using various techniques from simple batch processing to pipelining in distributed computer clusters. Various scripting languages, such as shell scripting, Perl, and R, are also very useful to devise pipelines. By pipelining, we can reduce total computing time and consequently increase throughput. In comparison to the traditional data processing pipeline residing on the central processors, performing general-purpose computation on a graphics processing unit provide a new-generation approach to massive parallel computing in genomic selection. While the concept of HTC may still be new to many researchers in animal breeding, plant breeding, and genetics, HTC infrastructures have already been built in many institutions, such as the University of Wisconsin–Madison, which can be leveraged for genomic selection, in terms of central processing unit capacity, network connectivity, storage availability, and middleware connectivity. Exploring existing HTC infrastructures as well as general-purpose computing environments will further expand our capability to meet increasing computing demands posed by unprecedented genomic data that we have today. We anticipate that HTC will impact genomic selection via better statistical models, faster solutions, and more competitive products (e.g., from design of marker panels to realized genetic gain). Eventually, HTC may change our view of data analysis as well as decision-making in the post-genomic era of selection programs in animals and plants, or in the study of complex diseases in humans.

Parallel Markov chain Monte Carlo - bridging the gap to high-performance Bayesian computation in animal breeding and genetics

“…The computing was implemented with a high-throughput computing pipeline called parallel-BayesCpC [30]. This is a high-throughput computing package and a member of the WGSE (Whole-Genome-enabled Selection and Evaluation) family [31] of distributed high-throughput computing pipelines.…”

“…The computing was implemented with a high-throughput computing pipeline called parallel-BayesCpC [30]. This is a high-throughput computing package and a member of the WGSE (Whole-Genome-enabled Selection and Evaluation) family [31] of distributed high-throughput computing pipelines. In computing, a pipeline is a set of data processing elements connected in series (i.e., the output of one element is the input of the next one) and the elements of a pipeline can be executed in parallel or sequentially.…”

“…This is especially true of Bayesian computation for genome-enabled prediction and selection, which aims at using whole-genome molecular data to predict the genetic merit of candidate animals for breeding purposes [1]. Typically, implementation of a high-dimensional model based on Markov chain Monte Carlo (MCMC) techniques is notoriously intensive in computing and often requires days, weeks, or even months of CPU (Central Processing Unit) time on personal computers and workstations [2]. Therefore, in order to overcome such computational burden, parallel computing becomes appealing [3,4].…”

“…Recently, parallel MCMC algorithms and strategies have become a focal point for scientific computing in the post-genome era [4]. This is largely due to the need to handle genomic datasets of unprecedented sizes, such as genome-wide dense markers or sequences for genome-enabled selection programs [2]. With a set of whole-genome markers (say 50K SNP markers or higher density) in a model, the computing task is highly challenging, particularly with sophisticated Bayesian models via MCMC implementation [1,11-13].…”

On measures of association among genetic variables

“…2004) or approximate Bayesian computation (Beaumont et al . 2002), can be implemented effectively in today′s high throughput systems (Wu et al . 2011).…”