Open problems in network-aware data management in exa-scale computing and terabit networking era

Balman, Mehmet; Byna, Suren

doi:10.1145/2110217.2110229

Cited by 6 publications

(7 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, the recent "Replica Core Archive" data from the IPCC Fifth Assessment Report (AR5) is expected to be around 2PB [10], whereas, the IPCC Forth Assessment Report (AR4) data archive is only 35TB. This trend can be seen across many areas in science [2,9].…”

Section: Optimizationsmentioning

confidence: 84%

See 1 more Smart Citation

Experiences with 100Gbps network applications

Balman

Pouyoul

Yao

et al. 2012

Proceedings of the Fifth International Workshop on Data-Intensive Distributed Computing Date

Self Cite

View full text Add to dashboard Cite

Abstract100Gbps networking has finally arrived, and many research and educational institutions have begun to deploy 100Gbps routers and services. ESnet and Internet2 worked together to make 100Gbps networks available to researchers at the Supercomputing 2011 conference in Seattle Washington. In this paper, we describe two of the first applications to take advantage of this network. We demonstrate a visualization application that enables remotely located scientists to gain insights from large datasets. We also demonstrate climate data movement and analysis over the 100Gbps network. We describe a number of application design issues and host tuning strategies necessary for enabling applications to scale to 100Gbps rates.

show abstract

Section: Optimizationsmentioning

confidence: 84%

“…Scientific applications for climate analysis are highly data-intensive [5,2,10,9]. A common approach is to stage data sets into local storage, and then run climate applications on the local data files.…”

Section: Climate Data Distribution Over 100gbpsmentioning

confidence: 99%

Experiences with 100Gbps network applications

Balman

Pouyoul

Yao

et al. 2012

Proceedings of the Fifth International Workshop on Data-Intensive Distributed Computing Date

Self Cite

View full text Add to dashboard Cite

show abstract

“…Main difference of our approach from these advance bandwidth reservation systems is that we set the scheduling goal to optimize resource usage to meet application requirements opportunistically. This means giving up on guaranteed service performance but enables us to design a system that performs 'well-enough', which is sufficient from users' point of view in most cases [3]. Furthermore, problem of providing bandwidth guarantees in shared transport networks is NP-hard as shown in [32].…”

Section: Related Workmentioning

confidence: 99%

“…The data transport medium is also typically shared by multiple application workflows with possibly competing application specific coupling requirements. Furthermore, best-effort solutions, which inherently can not offer service guarantees, will not be able to satisfy these requirements and achieve desired performance over utilized service medium [3].…”

Section: Introductionmentioning

confidence: 99%

Scheduling and flexible control of bandwidth and in-transit services for end-to-end application workflows

Aktas

Haldeman

Parashar

2016

Future Generation Computer Systems

View full text Add to dashboard Cite

Please cite this article as: M. Fatih Aktas, G. Haldeman, M. Parashar, Scheduling and flexible control of bandwidth and in-transit services for end-to-end application workflows, Future Generation Computer Systems (2015), http://dx. AbstractEnd-to-end scientific application workflows that integrate high-end experiments and instruments with large scale simulations and end-user displays are becoming increasingly important. These workflows require complex couplings and data sharing between distributed components involving large data volumes and present varying hard (in-time data delivery) and soft (in-transit processing) quality of service (QoS) requirements. As a result, supporting efficient data transport is critical for such workflows. In this paper, we leverage software-defined networking (SDN) to address issues of data transport service control and resource provisioning to meet varying QoS requirements from multiple coupled workflows sharing the same service medium. Specifically, we present a flexible control and a disciplined resource scheduling approach for data transport services for science networks. Furthermore, we emulate an SDN testbed on top of the FutureGrid virtualized testbed and use it to evaluate our approach for a realistic scientific workflow. Our results show that SDN-based control and resource scheduling based on simple intuitive models can meet the requirements of the targeted workflows with high resource utilization.

show abstract

“…In the age of high-bandwidth networks [11,10,15], challenges facing large-scale data transfer is a reality.…”

Section: Introductionmentioning

confidence: 99%

Advance Resource Provisioning in Bulk Data Scheduling

Balman

2013

2013 IEEE 27th International Conference on Advanced Information Networking and Applications (AINA)

View full text Add to dashboard Cite

Today's scientific and business applications generate massive data sets that need to be transferred to remote sites for sharing, processing, and long term storage. Because of increasing data volumes and enhancement in current network technology that provide ondemand high-speed data access between collaborating institutions, data handling and scheduling problems have reached a new scale. In this paper, we present a new data scheduling model with advance resource provisioning, in which data movement operations are defined with earliest start and latest completion times. We analyze timedependent resource assignment problem, and propose a new methodology to improve the current systems by allowing researchers and higher-level meta-schedulers to use data-placement as-a-service, so they can plan ahead and submit transfer requests in advance. In general, scheduling with time and resource conflicts is NP-hard. We introduce an efficient algorithm to organize multiple requests on the fly, while satisfying users' time and resource constraints. We successfully tested our algorithm in a simple benchmark simulator that we have developed, and demonstrated its performance with initial test results.

show abstract

Open problems in network-aware data management in exa-scale computing and terabit networking era

Cited by 6 publications

References 9 publications

Experiences with 100Gbps network applications

Experiences with 100Gbps network applications

Scheduling and flexible control of bandwidth and in-transit services for end-to-end application workflows

Advance Resource Provisioning in Bulk Data Scheduling

Contact Info

Product

Resources

About