Monadic composition for deterministic, parallel batch processing

Abstract

Achieving determinism on real software systems remains difficult. Even a batch-processing job, whose task is to map input bits to output bits, risks nondeterminism from thread scheduling, system calls, CPU instructions, and leakage of environmental information such as date or CPU model. In this work, we present a system for achieving low-overhead deterministic execution of batch-processing programs that read and write the file systemÐturning them into pure functions on files. We allow multi-process executions where a permissions system prevents races on the file system. Process separation enables different processes to enforce permissions and enforce determinism using distinct mechanisms. Our prototype, DetFlow, allows a statically-typed coordinator process to use shared-memory parallelism, as well as invoking process-trees of sandboxed legacy binaries. DetFlow currently implements the coordinator as a Haskell program with a restricted I/O type for its main function: a new monad we call DetIO. Legacy binaries launched by the coordinator run concurrently, but internally each process schedules threads sequentially, allowing dynamic determinism-enforcement with predictably low overhead. We evaluate DetFlow by applying it to bioinformatics data pipelines and software build systems. DetFlow enables determinizing these data-processing workflows by porting a small amount of code to become a statically-typed coordinator. This hybrid approach of static and dynamic determinism enforcement permits freedom where possible but restrictions where necessary. CCS Concepts: • Software and its engineering → Constraints; Concurrent programming structures;

Detection of river flow slow-down through sensing system and quasi-real time imaging

“…One of the examples is that they can be used to add or subtract a bias value to make a picture brighter or darker. The approach proposed in this paper is based on global operators [13] that create a destination pixel based on the entire image information and can be applied in image processing when the input is a set of pixels forming a 2D function that is already discrete. The output can then be expressed as follows:…”

Reproducible Containers

“…The scope of their guarantees is thus too small to be useful for reproducible builds. One exception is DetFlow [11] which provides deterministic parallel execution for batch jobs, though it lacks robust system call interception and requires a coordinator layer written in Haskell.…”

“…Determinism is useful in software builds [1,2], reproducible data analytics [3,4], and fault-tolerant distributed systems [5][6][7][8]. Yet in spite of previous work on deterministic languages [9][10][11] and operating systems [12][13][14], it is challenging to enforce deterministic output in practice. Thus we seek a practical container abstraction to isolate running software and execute it against clearly delimited input data, achieving end-to-end reproducible handling of data.…”

Metrology and Measurement Systems

“…A further comparison is required to strengthen the aforementiond results. We use a computer vision approach for robotics by implementing a processing of both images through a monadic algorithm [35] in the effort to extract features. A monadic operation allows to process the levels of intensity of pixels of original input image by producing one output image.…”