Dynamic Erasure Coding Policy Allocation (DECPA) in Hadoop 3.0

Tiny datablock in saving Hadoop distributed file system wasted memory

“…Lastly, the 225 MB wasted memory is one of three copies that HDFS must create in order of data high availability in case of data reading failure because of the unavailability of the desired data. Thus, a 225 MB will be turned into a 675 MB of total wasted memory [15]. These numbers shown a huge wasted memory amount just to host a small dataset [11].…”

“…Figure 2. HDSF replication manner [15] In Figure 2, the 1 Gb file requires approximately 16 Datablocks to store it. Then, to apply the HDFS high availability principle, all of these datablocks must be replicated 3 times in total.…”

Improving Performance and Capacity Utilization in Cloud Storage for Content Delivery and Sharing Services

“…Erasure Coding (EC) is a static redundancy strategy that offers similar levels of fault tolerance as replication, but with less storage space. This technique is used in Hadoop 3.0 [19] and Open Stack Swift [20]. Dynamic replication methods [21], [22] allows to adjust configurations on-the-fly during the storage service life cycle, based usually on monitoring data.…”

“…The main advantages of our proposal with respect to the related work are: i) It allows the creation of policies for providing cloud storage solution with data availability that improves storage utilization and performance of the information sharing for CDS services based on virtual machines and containers; ii) Most of the related work have been tested in simulated scenarios, while our methodology was tested in a prototype of a semantic CDS service for an organizational scenario; iii) To the best of our knowledge, this work would be the first approach that provides an integrated load balancing, availability and reliability scheme based on the classification of users and content (topics) activity, which dynamically adapt the replication factor and distribution of contents in a semantic CDS service to improve the trade-off between I/O performance and cloud storage consumption; and iv) A complementary simulator and emulator based on our methodology is available to evaluate state-of-the-art availability policies and load balancing methods at different scales. Some aspects that were not considered in our proposal and are available in the related work are: i) The use of synchronization techniques on the end-user side that can potentially improve user experience [32], [34]; and ii) The use of Erasure Coding (EC) techniques to provide data confidentiality and improve storage consumption [19], [20], [35].…”

Evaluating Erasure Codes in Dicoogle PACS

“…Moreover, previous versions of HDFS achieved fault tolerance by replicating multiple copies of data, such as RAID-1 on traditional storage arrays. Furthermore, the HDFS-EC (storage system implementing erasure code) system developed and implemented by Cloudera substantially reduces storage overhead while achieving similar or better fault tolerance by using parity cells such as RAID-5 [40].…”