Provision of Disk I/O Guarantee for MapReduce Applications

Tran, Xuan Thi; Do, Tien Van; H., Nam; Farkas, Lóránt; Rotter, Csaba

doi:10.1109/trustcom.2015.576

Cited by 4 publications

(8 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Motivated by the need, we design a complete solution that can be applied to provision the I/O data rate of applications in both the Mesos and YARN frameworks. Note that this is a result 1 that has gradually been improved over the years based on our previous experiences [10,11,24]. We demonstrate that the proposed functionalities can be integrated into two popular data processing frameworks such as Mesos and YARN to control the I/O data rates (disk I/O and network I/O) of applications, which may relieve the pain of service providers on the integration of schedulers to existing frameworks.…”

Section: Introductionmentioning

confidence: 74%

“…The idea of co-locating different scheduler frameworks in the same data center has been in discussion for the benefit of the operators and service providers. For example, only HDFS read traffic shaping in a YARN cluster using Traffic Control (LTC) mechanism was proposed in [11], while the disk I/O problem for Hadoop MapReduce applications and Spark applications [3,28] was investigated in [10,24]. Xu and Zhao [27] presented an interposed big-data I/O scheduler to provide I/O performance differentiation for competing applications in a shared big-data system.…”

Section: Related Workmentioning

confidence: 99%

“…Since a computing cluster can be built up of heterogeneous hardware and software components, the I/O data rate perceived by applications is unpredictable due to the contention for the resources of the physical servers, and there is a need to monitor applications running on these platforms as well [12]. As shown in [11,24], the I/O contention of applications leads to the degradation of quality of service.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Provisioning Input and Output Data Rates in Data Processing Frameworks

Farkas³

et al. 2020

J Grid Computing

Self Cite

View full text Add to dashboard Cite

This paper is motivated by the need of deadline-bounded applications in live mobile network environments to obtain the guarantee and the appropriate share of an input and output (I/O) data rate. However, data processing frameworks only support the request of memory and the computing capacity at present. In this paper, we propose a solution that allows the control of disk I/O and network I/O for data processing applications in YARN and Mesos frameworks. Experimental results show that our tool can provision the I/O data rate sharing of competing data processing applications.Keywords I/O enforcement · I/O data rate control · Cluster resource management · Hadoop YARN · Apache Mesos · HDFS T. V. Do ( )

show abstract

Section: Introductionmentioning

confidence: 74%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Provisioning Input and Output Data Rates in Data Processing Frameworks

Farkas³

et al. 2020

J Grid Computing

Self Cite

View full text Add to dashboard Cite

show abstract

“…Some customers may require a data rate guarantee because their jobs should be finished by a certain deadline. Therefore, the provision of the quality of service regarding a data rate guarantee may play a key factor to attract customers [69,70].…”

Section: Qos Guarantee In Cloud Environmentsmentioning

confidence: 99%

“…In recent collaboration works with Nokia [69,70], we proposed a set of functionality to monitor and isolate I/O demands in production environments. The proposed functionality can be used to minimize contention situations that lead to the I/O degradation offered to applications and clients.…”

Section: Qos Guarantee In Cloud Environmentsmentioning

confidence: 99%

BME VIK Annual Research Report on Electrical Engineering and Computer Science 2015

Vajk¹,

Harsányi

Poppe

et al. 2016

Period. Polytech. Elec. Eng. Comp. Sci.

View full text Add to dashboard Cite

PrefaceThroughout László Vajta, Dean of BME VIK János Levendovszky, Deputy Dean of BME VIK Department of Automation and Applied InformaticsDepartment of Automation and Applied Informatics (AUT) is one of the largest and dynamically developing departments at BME with diverse scope competences such as control theory, embedded systems, software modelling, applied software development, Internet of Things, power electronics, mechatronics, and many others. The department's main activities are education, research and development. Training versatile electrical and software engineers with solid practical knowledge is our top priority. Our profile also includes developing high quality software and hardware solutions for industry partners. All of our activities are backed by strong research background.In 2015, the department has organized the 21 st European Wireless (EW) Conference. The EW 2015 conference hosted more than 140 participants from 30 different countries. The 2015 edition of EW was aimed at addressing a key theme on "5G and beyond."The next sections summarize the key research results of the department in year 2015. IoT and Smart CityThe Internet of Things (IoT) is transforming the surrounding physical objects into an ecosystem of information that enriches our everyday life. The IoT represents the convergence of advances in miniaturization, wireless connectivity, and increased data storage, and is driven by various sensors. Application and service development methods and frameworks are required to support the realization of solutions covering data collection, transmission, data processing, analysis, reporting, and advanced querying. Our SensorHUB framework URBMOBI is a mobile environmental sensor that (i) provides temporally and spatially distributed environmental data, (ii) fulfills the need for monitoring at various places without the costs for a large number of fixed measurement stations, (iii) integrates small and precise sensors in a system that can be operated on buses, trams, or other vehicles, (iv) concentrates on urban heat and thermal comfort, and (v) aims at providing climate services and integration with real-time climate models.utilizes the state-of-the-art open source technologies and provides a unified tool chain for IoT related application and ser-URBMOBI project has been worked out between 2013 and 2015 by the following consortium: RWTH Aachen University (Germany), Netherlands Organization for Applied Scientific Research TNO (Netherlands), ARIA Technologies (France), Budapest University of Technology and Economics (Hungary), and Meteorological and Environmental Earth Observation (Italy).The SOLSUN (Sustainable Outdoor Lighting & Sensory Urban Networks) project is about to demonstrate how intelligent city infrastructure can be created in a cost-effective and sustainable way by reusing existing street lighting as the communications backbone. We apply different technologies and methods to reduce energy consumption at the same time as turning streetlights into nodes on a scalable network that is al...

show abstract

Weight‐based strategy for an I/O‐intensive application at a cloud data center

Peng

Liu

Wang

et al. 2018

Concurrency and Computation

View full text Add to dashboard Cite

Applications with different characteristics in the cloud may have different resources preferences.However, traditional resource allocation and scheduling strategies rarely take into account the characteristics of applications. Considering that an I/O-intensive application is a typical type of application and that frequent I/O accesses, especially small files randomly accessing the disk, may lead to an inefficient use of resources and reduce the quality of service (QoS) of applications, a weight allocation strategy is proposed based on the available resources that a physical server can provide as well as the characteristics of the applications. Using the weight obtained, a resource allocation and scheduling strategy is presented based on the specific application characteristics in the data center. Extensive experiments show that the strategy is correct and can guarantee a high concurrency of I/O per second (IOPS) in a cloud data center with high QoS. Additionally, the strategy can efficiently improve the utilization of the disk and resources of the data center without affecting the service quality of applications. KEYWORDS IOPS, process scheduling, random I/O, small files, weight INTRODUCTIONIn recent years, cloud computing has been widely used because of its advantages, such as its high scalability, low energy-consumption, and on-demand provision of virtual resources. Currently, companies such as Google, IBM, Amazon, Microsoft, and Alibaba are the leaders in the development and application of cloud computing technology and services. 1 As a new business mode, cloud computing provides computing, storage, and network services to Internet users by integrating resources. As a result, users conveniently make use of their applications with much lower purchase and maintenance costs. Additionally, cloud computing can greatly improve the resource utilization efficiency of society. Therefore, many enterprises as well as personal users have moved their applications to the cloud environment. In cloud computing, the data center is the key feature, as it owns a large amount of the resource pool that provides powerful resources and serves all types of applications.However, although the resources in a cloud data center are enormous, they are limited. Different consumptions of resources might considerably affect the performance and service capability of a cloud data center. Traditionally, resources are generally allocated based on a virtual machine (VM).Applications are executed on the virtual machine without considering their characteristics or features. Actually, different types of applications have different characteristics and consume different types and amounts of resources during their execution. These resource requirement difference among applications leads to serious problems for cloud resource allocations and task schedule strategies. For example, some types of resources may have been highly overloaded while other types of resources are barely consumed in different virtual machines or by different applications. This occur...

show abstract

Provision of Disk I/O Guarantee for MapReduce Applications

Cited by 4 publications

References 5 publications

Provisioning Input and Output Data Rates in Data Processing Frameworks

Provisioning Input and Output Data Rates in Data Processing Frameworks

BME VIK Annual Research Report on Electrical Engineering and Computer Science 2015

Weight‐based strategy for an I/O‐intensive application at a cloud data center

Contact Info

Product

Resources

About