Region templates: Data representation and management for high-throughput image analysis

Abstract

We introduce a region template abstraction and framework for the efficient storage, management and processing of common data types in analysis of large datasets of high resolution images on clusters of hybrid computing nodes. The region template abstraction provides a generic container template for common data structures, such as points, arrays, regions, and object sets, within a spatial and temporal bounding box. It allows for different data management strategies and I/O implementations, while providing a homogeneous, unified interface to applications for data storage and retrieval. A region template application is represented as a hierarchical dataflow in which each computing stage may be represented as another dataflow of finer-grain tasks. The execution of the application is coordinated by a runtime system that implements optimizations for hybrid machines, including performance-aware scheduling for maximizing the utilization of computing devices and techniques to reduce the impact of data transfers between CPUs and GPUs. An experimental evaluation on a state-of-the-art hybrid cluster using a microscopy imaging application shows that the abstraction adds negligible overhead (about 3%) and achieves good scalability and high data transfer rates. Optimizations in a high speed disk based storage implementation of the abstraction to support asynchronous data transfers and computation result in an application performance gain of about 1.13×. Finally, a processing rate of 11,730 4K×4K tiles per minute was achieved for the microscopy imaging application on a cluster with 100 nodes (300 GPUs and 1,200 CPU cores). This computation rate enables studies with very large datasets.

Parallel and Efficient Sensitivity Analysis of Microscopy Image Segmentation Workflows in Hybrid Systems

“…Our PATS mapping strategy uses variability in tasks performance to better utilize hybrid machines. In our earlier work [9], [24], we evaluated PATS using a two-stage application analysis pipeline consisting of segmentation and feature computation stages. In this work, the application consists of normalization, segmentation, and comparison stages.…”

“…The region templates (RT) framework supports the execution of dataflow applications [9]. Stages of a workflow in RT consume and produce region template data objects instead of reading/writing data directly from/to other stage or disk.…”

“…Thus, the tasks are likely to attain different speedups when executed on a co-processor. In order to take this performance variability into account, we developed the Performance-Aware Task Scheduling (PATS) [9]. PATS assigns tasks to a CPU core or an accelerator based on the tasks estimated acceleration on each device and on the device load.…”

Parallel A-Star Multiple Sequence Alignment with Locality-Sensitive Hash Functions

“…In this work, we used two H functions: the SU M function, that simply adds all numbers in the coordinate; and the ZORDER function, which is based on Z-Order curves [14]. The ZORDER function was selected because it preserves locality and is inexpensive, because it consists of shift operations to interleave the bits of numbers in the coordinate values.…”

“…During the expand phase, a locally partial hash function is used to determine in which list the neighbor node should be inserted (line 14). Nodes that belong to thread i are inserted in queue q i .…”