An MPI Implementation of a Self-Submitting Parallel Job Queue

Abstract

Abstract. We present a simple and easy to apply methodology for using high-level self-submitting parallel job queues in an MPI environment. Using C++, we implemented a library of functions, MPQueue, both for testing our concepts and for use in real applications. In particular, we have applied our ideas toward solving computational combinatorics problems and for finding bifurcation diagrams of solutions of partial differential equations (PDE). Our method is general and can be applied in many situations without a lot of programming effort. The key idea is that workers themselves can easily submit new jobs to the currently running job queue. Our applications involve complicated data structures, so we employ serialization to allow data to be effortlessly passed between nodes. Using our library, one can solve large problems in parallel without being an expert in MPI. We demonstrate our methodology and the features of the library with several example programs, and give some results from our current PDE research. We show that our techniques are efficient and effective via overhead and scaling experiments.

Newton's Method and Symmetry for Semilinear Elliptic PDE on the Cube

“…The symmetry of the reduced vector fieldg onẼ for this bifurcation with T d symmetry is shown in Figure 3 Using these MI values, we can make an index theory computation (15) to verify that the results are consistent with having obtained all solutions.…”

“…A serial implementation could not reproduce our results in a reasonable time. To that end, in [15] we developed a simple and easy to apply methodology for using high-level, self-submitting parallel job queues in an MPI (Message Passing Interface [6]) environment. In that paper, we apply our parallel job queue techniques toward solving computational combinatorics problems, as well as provide the necessary details for implementing our PDE algorithms in parallel C++ code in order to obtain the results found in the current article.…”

“…It was necessary to implement our code in parallel in order to get timely results. We use a library of functions (MPQueue) presented in [15] which make it easy to create and manage a parallel job queue. For the current application of creating a bifurcation diagram, we choose a natural way to decompose the task into two types of jobs, namely branch following and find daughters jobs.…”

“…This 2-day run used 24 processors and generated all the data necessary for Figures 14, 15, 20, 21, and 22. In Table 6.1 of [15] we provide runtime data for a similar problem using a varying number of processors, in order to demonstrate scalability.…”

“…Even with these efficiency improvements, we found it necessary to convert our high-level branch following strategy to use parallel computing. Most of the details of the parallel implementation can be found in [15], where we present a general methodology using self-submitting job queues to implement many types of mathematical algorithms in parallel. In particular, therein we develop a light-weight, easy-to-use C++ parallel job queue library, which we have used in obtaining the cube results found in this article.…”