Making work queue cluster-friendly for data intensive scientific applications

Abstract

Abstract-Researchers with large-scale data-intensive applications often wish to scale up applications to run on multiple clusters, employing a middleware layer for resource management across clusters. However, at the very largest scales, such middleware is often "unfriendly" to individual clusters, which are usually designed to support communication within the cluster, not outside of it. To address this problem we have modified the Work Queue master-worker application framework to support a hierarchical configuration that more closely matches the physical architecture of existing clusters. Using a synthetic application we explore the properties of the system and evaluate its performance under multiple configurations, with varying worker reliability, network capabilities, and data requirements. We show that by matching the software and hardware architectures more closely we can gain both a modest improvement in runtime and a dramatic reduction in network footprint at the master. We then run a scalable molecular dynamics application (AWE) to examine the impact of hierarchy on performance, cost and efficiency for real scientific applications and see a 96% reduction in network footprint, making it much more palatable to system operators and opening the possibility of increasing the application scale by another order of magnitude or more.

CoGe LoadExp+: A web‐based suite that integrates next‐generation sequencing data analysis workflows and visualization

“…Analyses are dispatched by a job scheduling engine, based on WorkQueue (Albrecht, Rajan, & Thain, ). Users can request a notification email upon completion and monitor the job under the Activity and Data Loading tabs in the My Data page, or with a persistent link displayed on the progress window.…”

United polarizable multipole water model for molecular mechanics simulation

“…65 At each optimization step, the set of 36 simulations at different phase points (32 temperatures at 1.0 atm pressure plus 4 pressures at 298.15 K temperature) is performed simultaneously on multiple nodes in a GPU cluster; the Work Queue library [66][67][68] allows ForceBalance to act as a distributed computing server and coordinate many OpenMM simulations running on multiple compute nodes in different physical locations. Finally, the data from the finished simulations were analyzed using the multistate Bennett acceptance ratio (MBAR) estimator, 69,70 which allows each simulation to contribute to the estimated properties of all other simulations.…”

“…Due to the non-overlapping features of the simulation codes, we combined OpenMM 6.1 and TINKER 6.3 during the optimization to evaluate quantities for comparison with the ab initio and gas phase reference data, using OpenMM to evaluate the potential energies and forces, and TINKER to evaluate the binding energies and monomer properties. ForceBalance, 71 TINKER, 65 OpenMM, 64 and Work Queue 68 are freely available on the web.…”

Exploiting volatile opportunistic computing resources with Lobster

“…To begin a project, the user starts the Lobster master, which divides the project submitted by the user into a series of tasks. We use Work Queue (WQ) [2], which consists of a library for creating master-worker style programs to manage the queue of tasks and the pool of workers. WQ assigns tasks to workers, notifies the master as tasks are completed, and assigns new tasks to workers as they become available.…”