Skyport - Container-Based Execution Environment Management for Multi-cloud Scientific Workflows

Gerlach, Wolfgang; Tang, Wei; Keegan, Kevin P.; Harrison, Travis; Wilke, Andreas; Bischof, Jared; D’Souza, Mark; Devoid, Scott; Murphy-Olson, Daniel E.; Desai, Narayan; Meyer, Folker

doi:10.1109/datacloud.2014.6

Cited by 84 publications

(35 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…[19]) and to deploy large scale big data applications, requiring high elasticity in managing a very large amount of concurrent components (e.g. [7,15,18]). …”

Section: Introductionmentioning

confidence: 99%

Measuring Docker Performance

Casalicchio

Perciballi

2017

Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering Companion

View full text Add to dashboard Cite

Today, a new technology is going to change the way platforms for the internet of services are designed and managed. This technology is called container (e.g. Docker and LXC). The internet of service industry is adopting the container technology both for internal usage and as commercial offering. The use of container as base technology for largescale systems opens many challenges in the area of resource management at run-time, for example: autoscaling, optimal deployment and monitoring. Specifically, monitoring of container based systems is at the ground of any resource management solution, and it is the focus of this work. This paper explores the tools available to measure the performance of Docker from the perspective of the host operating system and of the virtualization environment, and it provides a characterization of the CPU and disk I/O overhead introduced by containers.

show abstract

“…[19]) and to deploy large scale big data applications, requiring high elasticity in managing a very large amount of concurrent components (e.g. [7,15,18]). …”

Section: Introductionmentioning

confidence: 99%

Measuring Docker Performance

Casalicchio

Perciballi

2017

Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering Companion

View full text Add to dashboard Cite

show abstract

“…Thus, the effective isolation of each applications runtime environment becomes a challenging issue. As stated by [10] this problem can be partially solved with the use of customized Virtual Machines, created from pre-existing images, that are configured in advance with the required tools and dependencies. However, virtualization with hypervisors introduces a not negligible overhead on resources utilization [5] and also the boot process of VMs affects the overall latency of the system.…”

Section: A State Of the Artmentioning

confidence: 99%

Container-Based Orchestration in Cloud: State of the Art and Challenges

Tosatto

Ruiu

Attanasio

2015

2015 Ninth International Conference on Complex, Intelligent, and Software Intensive Systems

View full text Add to dashboard Cite

How to effectively manage increasingly complex enterprise computing environments is one of the hardest challenges that most organizations have to face in the era of cloud computing, big data and IoT. Advanced automation and orchestration systems are the most valuable solutions helping IT staff to handle large-scale cloud data centers. Containers are the new revolution in the cloud computing world, they are more lightweight than VMs, and can radically decrease both the startup time of instances and the processing and storage overhead with respect to traditional VMs. The aim of this paper is to provide a comprehensive description of cloud orchestration approaches with containers, analyzing current research efforts, existing solutions and presenting issues and challenges facing this topic.

show abstract

“…Developers can describe a data analysis pipeline in YAML or JSON files that contain a clear description of tools, input parameters, input and output data, and how the tools are connected. There are multiple systems that implement the CWL standard, including Galaxy [13], Toil, Arvados (https://arvados.org/), and AWE [29], making it possible to write a single description of a pipeline and run it in the most suitable pipeline execution environment. It is an open challenge to implement support for the standardized pipeline descriptions on execution environments such as Spark.…”

Section: Current Trendsmentioning

confidence: 99%

A Review of Scalable Bioinformatics Pipelines

Fjukstad

Bongo

2017

Data Sci. Eng.

View full text Add to dashboard Cite

Scalability is increasingly important for bioinformatics analysis services, since these must handle larger datasets, more jobs, and more users. The pipelines used to implement analyses must therefore scale with respect to the resources on a single compute node, the number of nodes on a cluster, and also to cost-performance. Here, we survey several scalable bioinformatics pipelines and compare their design and their use of underlying frameworks and infrastructures. We also discuss current trends for bioinformatics pipeline development.

show abstract

Skyport - Container-Based Execution Environment Management for Multi-cloud Scientific Workflows

Cited by 84 publications

References 19 publications

Measuring Docker Performance

Measuring Docker Performance

Container-Based Orchestration in Cloud: State of the Art and Challenges

A Review of Scalable Bioinformatics Pipelines

Contact Info

Product

Resources

About