Critical Study of Performance Parameters on Distributed File Systems Using MapReduce

Vaidya, Madhavi; Deshpande, Shrinivas

doi:10.1016/j.procs.2016.02.037

Cited by 10 publications

(10 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The three regional networks are connected to UNSDGs1-6N. In every grid network, there is a distributed file system, a Hadoop master server, load balancer, file server, and three Hadoop application servers [35]. As shown in Figure 4 above, the DNNSDGs1-6 is expected to have a handshake with external systems like social media (Facebook, Twitter, GooglePlus, Instagram, YouTube), email systems and other sustainable development information systems.…”

Section: Sustainability 2020 12 X For Peer Review 8 Of 16mentioning

confidence: 99%

A Deep Neural Network-Based Advisory Framework for Attainment of Sustainable Development Goals 1-6

et al. 2020

View full text Add to dashboard Cite

Research in sustainable development, program design and monitoring, and evaluation requires data analytics for the Sustainable Developments Goals (SDGs) not to suffer the same fate as the Millennium Development Goals (MDGs). The MDGs were poorly implemented, particularly in developing countries. In the SDGs dispensation, there is a huge amount of development-related data that needs to be harnessed using predictive analytics models such as deep neural networks for timely and unbiased information. The SDGs aim at improving the lives of citizens globally. However, the first six SDGs (SDGs 1-6) are more relevant to developing economies than developed economies. This is because low-resourced countries are still battling with extreme poverty and unacceptable levels of illiteracy occasioned by corruption and poor leadership. Inclusive innovation is a philosophy of SDGs as no one should be left behind in the global economy. The focus of this study is the implementation of SDGs 1-6 in less developed countries. Given their peculiar socio-economic challenges, we proposed a design for a low-budget deep neural network-based sustainable development goals 1-6 (DNNSDGs 1-6) system. The aim is to empower actors implementing SDGs in developing countries with data-based information for robust decision making.

show abstract

Section: Sustainability 2020 12 X For Peer Review 8 Of 16mentioning

confidence: 99%

A Deep Neural Network-Based Advisory Framework for Attainment of Sustainable Development Goals 1-6

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Selain Ceph, dalam virtualisasi telah banyak digunakan beragam sistem penyimpanan lain, seperti ZFS, GlusterFs, HDFS, dan Lustre [8]. Sistem file ZFS dikembangkan oleh Oracle sebagai pengganti file system tradisional dan volume manager yang terdapat dalam sebagian besar sistem operasi berbasis Linux atau Unix.…”

Section: Pendahuluanunclassified

“…Uji perbandingan kinerja antar sistem file juga telah dilakukan. [8] membandingkan kinerja beragam sistem file terdistribusi menggunakan map reduce, yaitu Glusterfs, Lustre, Ceph dan HDFS. [11] membandingkan kinerja sistem file GlusterFS dan Ceph serta [12] membandingkan ZFS dan LVM di lingkungan virtual.…”

Section: Pendahuluanunclassified

“…Berbeda dengan [8] yang menggunakan tolak ukur map reduce serta [11] dan [12] Kinerja throughput Ceph yang dihasilkan lebih rendah daripada [11] yang menguji kinerja Ceph pada VM menggunakan virtio sebagai driver KVM dengan penyimpanan terdistribusi. Nilai throughput pembacaan Ceph yang diperoleh adalah sebesar 60 MB/s, sedangkan untuk throughput penulisan sebesar 38 MB/s.…”

Section: Pendahuluanunclassified

See 1 more Smart Citation

Perbandingan Kinerja Block Storage Ceph dan ZFS di Lingkungan Virtual

Robbi

Prasetijo

Widianto

2019

Jurnal Teknologi dan Sistem Komputer

View full text Add to dashboard Cite

The growth of data requires better performance in the storage system. This study aims to analyze the comparison of block storage performance of Ceph and ZFS running in virtual environments. Tests were conducted to measure their performances, including IOPS, CPU usage, throughput, OLTP Database, replication time, and data integrity. Testing was done using 2 node servers with a standard configuration of the storage system. Server virtualization uses Proxmox on each node. ZFS has a higher performance of reading and writing operation than Ceph in IOPS, CPU usage, throughput, OLTP and data replication duration, except the CPU usage in writing operation. The test results are expected to be a reference in the selection of storage systems for data center applications.

show abstract

“…There are many of such file systems and the decision should not be taken lightly as different implementations compromise upon different tradeoffs, which will have a substantial effect on many aspects of the system performance. () For instance, in the context of high performance computing, Lustre and IBM's General Parallel File System (GPFS) are the most common, mainly thanks to their scalability and high performance characteristics. Other file systems are more focused on using commodity hardware, for instance, in the context of datacenters without RAID disks or a Storage Area Network (SAN).…”

Section: Introductionmentioning

confidence: 99%

On the effects of allocation strategies for exascale computing systems with distributed storage and unified interconnects

Pascual

Lant

Concatto

et al. 2018

Concurrency and Computation

View full text Add to dashboard Cite

The convergence between computing-and data-centric workloads and platforms is imposing new challenges on how to best use the resources of modern computing systems. In this paper, we investigate alternatives for the storage subsystem of a novel exascale-capable system with special emphasis on how allocation strategies would affect the overall performance. We consider several aspects of data-aware allocation such as the effect of spatial and temporal locality, the affinity of data to storage sources, and the network-level traffic prioritization for different types of flows.In our experimental set-up, temporal locality can have a substantial effect on application runtime (up to a 10% reduction), whereas spatial locality can be even more significant (up to one order of magnitude faster with perfect locality). The use of structured access patterns to the data and the allocation of bandwidth at the network level can also have a significant impact (up to 20% and 17% reduction of runtime, respectively). These results suggest that scheduling policies exposing data-locality information can be essential for the appropriate utilization of future large-scale systems. Finally, we found that the distributed storage system we are implementing can outperform traditional SAN architectures, even with a much smaller (in terms of I/O servers) back-end. KEYWORDSinter-processor communications, near-data computing, resource allocation, scheduling, storage traffic INTRODUCTIONTraditionally, supercomputers have been used to execute large computing-intensive parallel applications such as scientific codes. However, nowadays, new types of data-oriented applications are becoming increasingly popular. In contrast with traditional high performance computing (HPC) codes, they have to process massive amounts of scientific or business-oriented data and, hence, impose completely different needs to the computing systems.Indeed, new hardware and software are being developed to suit these necessities. One of these systems is our novel, custom-made architecture,ExaNeSt. 1 We are working on the design and construction of a prototype capable of reaching Exascale computation using tens of millions of interconnected low-power-consumption ARM cores. 2 To support such kind of data-intensive applications, we are leveraging a unified, low-latency Interconnection Network (hereafter, IN) and a fully distributed storage subsystem, BeeGFS, with the data spread across the nodes in local non-volatile storage devices 3 (NVM). This greatly contrasts with traditional supercomputers and datacenters that rely on Storage Area Networks (SAN) to access the data with separate networks for I/O, system management, and inter-processor communications (IPC).A fully distributed file system allows for near-data computation, reducing the great overheads of moving data from the centralized storage to the compute nodes. 4 A single, consolidated IN offers enormous power-savings when compared with multi-network designs. While a unified IN does, indeed, allow us to cope with power and co...

show abstract

Critical Study of Performance Parameters on Distributed File Systems Using MapReduce

Cited by 10 publications

References 9 publications

A Deep Neural Network-Based Advisory Framework for Attainment of Sustainable Development Goals 1-6

A Deep Neural Network-Based Advisory Framework for Attainment of Sustainable Development Goals 1-6

Perbandingan Kinerja Block Storage Ceph dan ZFS di Lingkungan Virtual

On the effects of allocation strategies for exascale computing systems with distributed storage and unified interconnects

Contact Info

Product

Resources

About