Kulla, a container-centric construction model for building infrastructure-agnostic distributed and parallel applications

Kulla‐RIV: A composing model with integrity verification for efficient and reliable data processing services

“…These interfaces can be set up as a file system (e.g., where the data source and data sink reside in a local file system or Docker volume) or as a network interface (e.g., establishing a connection between the data source and data sink through a Socket port or REST‐API provided by a storage service located at a specific URL). For a more in‐depth understanding of the Kulla‐block input/output interfaces, please refer to the extended explanation in the Kulla model 8 …”

“…The use of composition models (e.g., Kulla 8 ), workflows engines (e.g., Makeflow, 24 Parsl, 33 or DagOn* 34 ), and CI/CD pipelines manager (e.g., Jenkins 35 ) for the composition of distributed solutions has allowed developers to produce large‐scale applications across different IT infrastructures. These tools allow to chain and distribute applications in cloud environments and, in some cases, allow the use of parallelism techniques (task‐based or data‐based parallelism).…”

“…This file (known as DAG configuration file) describes all the processes interacting inside a Kulla‐Box, forming a DAG structure. Inside the Kulla‐Box, the Engine adapter supervises the execution of all involved applications following the extraction, transform, and load (ETL) model 8 . The incoming data is received by the input interface of the Kulla structure (extraction), then all the processes described in the DAG configuration file are executed (transform) and the results are finally sent to the next Kulla structure by the output interface (load).…”

“…Each processing stage (applications or services) can be implemented, deployed, and executed by one single organization, a set of partners, or even end‐users by using software‐oriented architectures or microservices architectures 6,7 . In this type of scenario, the organizations not only develop their own required applications to process their data but they could also need to integrate, into their systems, external applications or services produced by any of partners, external developers or end‐users, or cloud service providers 8,9 …”

“…The rest of the article is organized as follows. Section 2 summarizes the fundamentals of the Kulla model 8 . The design of the reliability and integrity verification scheme (RIV) is presented in Section 3.…”