An Efficient Indexing Mechanism for Data Deduplication

Thwel, Tin Thein; Thein, Ni Lar

doi:10.1109/ctit.2009.5423123

Cited by 21 publications

(12 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For this purpose, we will be using the B+ trees. B+ trees are an effective data structure tool for indexing the data for quick search and retrieval [22]. They were first proposed by Thwel and Thein [22] for indexing the data in the standard disk-based system, but their proposed approach did not considered the security requirement of the chunked data.…”

Section: Sdd Frameworkmentioning

confidence: 99%

“…B+ trees are an effective data structure tool for indexing the data for quick search and retrieval [22]. They were first proposed by Thwel and Thein [22] for indexing the data in the standard disk-based system, but their proposed approach did not considered the security requirement of the chunked data. We believe they can be used in cloud environments as well, for the purpose of indexing the data for quick search and retrieval of files by the CSP.…”

Section: Sdd Frameworkmentioning

confidence: 99%

“…Third, an indexing scheme is designed to generate an index of the encrypted file chunks for fast search purpose by the CSP. To this end, B+ trees [22] are used due to their efficiency as data structure tool for indexing the data for quick search and retrieval. Fourth, an asymmetric searchable encryption scheme [27] is used for encrypting the generated index of the reduplicated data.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Secure Data Deduplication in Cloud Environments

Rashid¹

2021

Preprint

View full text Add to dashboard Cite

With the tremendous growth of available digital data, the use of Cloud Service Providers (CSPs) are gaining more popularity, since these types of services promise to provide convenient and efficient storage services to end-users by taking advantage of a new set of benefits and savings offered by cloud technologies in terms of computational, storage, bandwidth, and transmission costs. In order to achieve savings in storage, CSPs often employ data dedplication techniques to eliminate duplicated data. However, benefits gained through these techniques have to balanced against users' privacy concerns, as these techniques typically require full access to data. In this thesis, we propose solutions for different data types (text, image and video) for secure data deduplication in cloud environments. Our schemes allow users to upload their data in a secure and efficient manner such that neither a semi-honest CSP nor a malicious user can access or compromise the security of the data. We use different image and video processing techniques, such as data compression, in order to further improve the efficiency of our proposed schemes. The security of the deduplication schemes is provided by applying suitable encryption schemes and error correcting codes. Moreover, we propose proof of storage protocols including Proof of Retrievability (POR) and Proof of Ownership (POW) so that users of cloud storage services are able to ensure that their data has been saved in the cloud without tampering or manipulation. Experimental results are provided to validate the effectiveness of the proposed schemes.

show abstract

Section: Sdd Frameworkmentioning

confidence: 99%

Section: Sdd Frameworkmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Secure Data Deduplication in Cloud Environments

Rashid¹

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In Reference [21] shows that, proposed algorithm for an efficient indexing mechanism using the advantage of B+ tree properties. In this algorithm, File is divided into variable-length chunks using Two Thresholds Two Divisors chunking algorithm.…”

Section: Literature Reviewmentioning

confidence: 99%

Efficient Cross User Client Side Data Deduplication in Hadoop

Prajapati¹,

Shah²,

Ganatra³

et al. 2017

JCP

View full text Add to dashboard Cite

Hadoop is widely used for applications like Aadhaar card, Healthcare, Media, Ad Platform, Fraud Detection & Crime, and Education etc. However, it does not provide efficient and optimized data storage solution. One interesting thing we found that when user uploads the same file twice with same file name it doesn't allow saving the same file. But when user uploads the same file content with different file name Hadoop allows uploading that file. In general same files are uploaded by many users (cross user) with different name with same contents so this leads to wastage of storage space. So we provided the solution of above problem and provide Data Deduplication in Hadoop. Before uploading data to HDFS we calculate Hash Value of File and stored that Hash Value in Database for later use. Now same or other user wants to upload the same content file but with same content, our DeDup module will calculate Hash value and verify it to HBase. Now if Hash Value is matched so it will give message that "File is already exits". Experimental analysis demonstrates (i.e. Text, Audio, Video, Zip files etc.) that proposed solution gives more optimized storage acquiring very small computation overhead and having optimized storage space.

show abstract

“…AA-Dedupe [4] analyzes the characteristics of the files and apply a suitable chunking mechanism to minimize the metadata overhead. Efficient indexing [5] and scalable hybrid hash cluster [6] have utilized efficient data structures to maintain the metadata. Guo et al [7] and Efstathopoulos et al [8] used a sampling technique to obtain a sample subset of metadata from the disk to improve the read throughput.…”

Section: Related Workmentioning

confidence: 99%

Concurrency control algorithms for deduplicated cloud storage

Balasundaram¹,

Babu²

2018

Turk J Elec Eng & Comp Sci

View full text Add to dashboard Cite

Deduplication of data is essential to effectively use cloud storage. As the metadata in deduplicated cloud storage are shared across multiple users, concurrent updates may result in inconsistencies. A coarse-grained locking strategy that has been proposed earlier to overcome this difficulty is not suited for inline deduplication owing to poor performance. In the present work, a fine-grained locking strategy that overcomes this shortcoming is proposed. A metadata structure along with a set of concurrent control mechanisms to accomplish this is presented. This strategy is shown to improve the throughput by as much as 60% with only marginal lock overhead.

show abstract

An Efficient Indexing Mechanism for Data Deduplication

Cited by 21 publications

References 2 publications

Secure Data Deduplication in Cloud Environments

Secure Data Deduplication in Cloud Environments

Efficient Cross User Client Side Data Deduplication in Hadoop

Concurrency control algorithms for deduplicated cloud storage

Contact Info

Product

Resources

About