3DNBS: A Data De-duplication Disk-Based Network Backup System

Abstract

Traditionally, backup and archiving have been performed on tapes. With the rapid advances in disk storage technology witnessed in recent years, it becomes practical to use disks other than tape libraries as backend storage device for a backup system. For such a disk-based system, storage space efficiency is essential. Since traditional backup method cannot eliminate redundancies during backup, a new data deduplication backup technique should be developed to provide more efficient data storage at the system. This paper describes the design and performance evaluation of a data de-duplication disk-based network backup system, called 3DNBS. 3DNBS breaks files into variable sized chunks using content-defined chunking (CDC) for the purpose of duplication detection. Chunks are indexed and addressed by hashing their content, which leads to intrinsically single instance storage. Experimental results show that in comparison with traditional backup method such as Bacula, 3DNBS presents dramatic reduction in required storage space on various workloads. By eliminating duplicated data, 3DNBS also reduces the size of data to be transmitted, hence reducing time to perform backup in a bandwidth constraint environment.

A Cluster-Based Data De-duplication Technology

“…In order to speed up the comparison time, [8] [13]. The hierarchical index table used few chunk IDs to represent files and to store them in memory.…”

Research on a Clustering Data De-Duplication Mechanism Based on Bloom Filter

“…However, LBFS [7], fingerdiff [10], REBL [11], 3DNBS [12] and SDD [13] used hash algorithm (SHA-1 or MD5), and most of them considered that the probability of hash collision is extremely lower than the probability of hardware errors. This paper selects SHA-1 hash algorithm because of its strong collision resistant and encryption function.…”