On the use of microservers in supporting hadoop applications

Anwar, Ali; Krish, K. R.; Butt, Ali R.

doi:10.1109/cluster.2014.6968753

Cited by 16 publications

(9 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Awasthi et al [12] also perform a system level characterization of data center applications. Anwar et al [11] conduct a quantitative study of representative Hadoop applications on five hardware configurations with the purpose of evaluating the different clusters' performance. The state-of-the-art work of characterizing scale-out (data center) workloads on a micro-architecture level is Cloud-Suite [17].…”

Section: Related Workmentioning

confidence: 99%

Understanding Big Data Analytics Workloads on Modern Processors

Jia

Zhan

Wang

et al. 2017

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

Big data analytics applications play a significant role in data centers, and hence it has become increasingly important to understand their behaviors in order to further improve the performance of data center computer systems, in which characterizing representative workloads is a key practical problem. In this paper, after investigating three most important application domains in terms of page views and daily visitors, we chose 11 representative data analytics workloads and characterized their micro-architectural behaviors by using hardware performance counters, so as to understand the impacts and implications of data analytics workloads on the systems equipped with modern superscalar out-of-order processors. Our study reveals that big data analytics applications themselves share many inherent characteristics, which place them in a different class from traditional workloads and scale-out services. To further understand the characteristics of big data analytics workloads we performed a correlation analysis of CPI (cycles per instruction) with other microarchitecture level characteristics and an investigation of the big data software stack impacts on application behaviors. Our correlation analysis showed that even though big data analytics workloads own notable pipeline front end stalls, the main factors affecting the CPI performance are long latency data accesses rather than the front end stalls. Our software stack investigation found that the typical big data software stack significantly contributes to the front end stalls and incurs bigger working set. Finally we gave several recommendations for architects, programmers and big data system designers with the knowledge acquired from this paper.

show abstract

Section: Related Workmentioning

confidence: 99%

Understanding Big Data Analytics Workloads on Modern Processors

Jia

Zhan

Wang

et al. 2017

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

show abstract

“…MOS has a wider scope -as a cloud object store, it provides fault tolerance functionality; more importantly, it aims to improve the overall performance by efficiently exploiting the data center and workload heterogeneity. [11] proposes a metric based on CPU, I/O wait and memory usage that are critical for Hadoop. Similarly, MET [17] proposes several system metrics that are critical for a NoSQL database and highly impacts server utilization's estimation.…”

Section: Related Workmentioning

confidence: 99%

Taming the cloud object storage with MOS

Anwar

Cheng

Gupta

et al. 2015

Proceedings of the 10th Parallel Data Storage Workshop

Self Cite

View full text Add to dashboard Cite

Cloud object stores today are deployed using a single set of configuration parameters for all different types of applications. This homogeneous setup results in all applications experiencing the same service level (e.g., data transfer throughput, etc.). However, the vast variety of applications expose extremely different latency and throughput requirements. To this end, we propose MOS, a Micro Object Storage architecture with independently configured microstores each tuned dynamically for a particular type of workload. We then expose these microstores to the tenant who can then choose to place their data in the appropriate microstore according the latency and throughput requirements of their workloads. Our evaluation shows that compared with default setup, MOS can improve the performance up to 200% for small objects and 28% for large objects while providing opportunity of tradeoff between two.

show abstract

“…Centrally managed HPC clusters at Virginia Tech typically run batch jobs in Portable Batch System (PBS) executed by a Moab workload manager with resources allocation controlled by a TORQUE resource manager 1 . e managed HPC clusters come with a variety of so ware modules (e.g.…”

Section: Introductionmentioning

confidence: 99%

Spark on the ARC

DeYoung

Salman

Bedi

et al. 2017

Practice and Experience in Advanced Research Computing 2017: Sustainability, Success and Impact

View full text Add to dashboard Cite

In this paper we document our approach to overcoming service discovery and con guration of Apache Hadoop and Spark frameworks with dynamic resource allocations in a batch oriented Advanced Research Computing (ARC) High Performance Computing (HPC) environment. ARC e orts have produced a wide variety of HPC architectures. A common HPC architectural pa ern is multi-node compute clusters with low-latency, high-performance interconnect fabrics and shared central storage. is pa ern enables processing of workloads with high data co-dependency, frequently solved with message passing interface (MPI) programming models, and then executed as batch jobs. Unfortunately, many HPC programming paradigms are not well suited to big data workloads which are o en easily separable. Our approach lowers barriers of entry to HPC environments by enabling end users to utilize Apache Hadoop and Spark frameworks that support big data oriented programming paradigms appropriate for separable workloads in batch oriented HPC environments. CCS CONCEPTS •Information systems → Computing platforms; Nearest-neighbor search; •So ware and its engineering → So ware as a service orchestration system;

show abstract

On the use of microservers in supporting hadoop applications

Cited by 16 publications

References 28 publications

Understanding Big Data Analytics Workloads on Modern Processors

Understanding Big Data Analytics Workloads on Modern Processors

Taming the cloud object storage with MOS

Spark on the ARC

Contact Info

Product

Resources

About