On Protocol-Independent Data Redundancy Elimination

Zhang, Yan; Ansari, Nirwan

doi:10.1109/surv.2013.052213.00186

Cited by 25 publications

(6 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We adopt a content-defined chunking algorithm for the first time to convert a sequence-type feature into a set for the purpose of indexing and search of malicious PE files [15]. We choose an asymmetric extremum (AE) chunking algorithm because this runs quickly and divides any sequence into evenly-sized chunks around the average chunk size [16].…”

Section: B Feature Set Convertermentioning

confidence: 99%

Scalable and Multifaceted Search and Its Application for Binary Malware Files

2021

View full text Add to dashboard Cite

Malicious binary files are a serious threat to industrial information systems. Because of their large number, an automatic assistant tool becomes essential for analysis, and finding similar files would be a great help. In this paper, we present a fast, scalable, and multifaceted search scheme to find similar binary malware files. We use a content-defined chunking algorithm to convert a file into a feature set for the first time. The proposed scheme uses minHash to reduce any feature set of any file to a fixed size, which significantly improves search accuracy, processing speed, and space utilization. We theoretically prove that the new scheme returns similar files in jaccard index order. Through implementation and experiments with 12 million malicious files, we confirm that the search speed is increased by 600%, space is reduced by 90%, and the accuracy is increased by 400% at least, compared with the state-of-the-art of Elasticsearch.

show abstract

Section: B Feature Set Convertermentioning

confidence: 99%

Scalable and Multifaceted Search and Its Application for Binary Malware Files

2021

View full text Add to dashboard Cite

show abstract

“…Redundant data consumes the network resources and deteriorate the network performance by increasing the congestion. Due to the rapid increase of internet data, many data redundancy techniques have been introduced in recent years [12,13,14]. Many current techniques provided suitable solutions to improve the network performance by removing the data redundancy in the network.…”

Section: Related Work and Motivationsmentioning

confidence: 99%

“…It has been broadly agreed that data redundancy eradication offers great benefits in practice. Generally, the benefit of removing the data redundancy is the improved network performance in terms of higher network throughput and lower end to end delay [12,13,14]. However, the currently provided solutions are not so effective and remove a part of the original data with redundant data.…”

Section: Related Work and Motivationsmentioning

confidence: 99%

See 1 more Smart Citation

An Energy-Efficient Redundant Transmission Control Clustering Approach for Underwater Acoustic Networks

Zhao

Fareed

2019

Sensors

View full text Add to dashboard Cite

Underwater Acoustic Network (UAN) is an emerging technology with attractive applications. In such type of networks, the control-overhead, redundant inner-network transmissions management, and data-similarity are still very challenging. The cluster-based frameworks manage the control-overhead and redundant inner-network transmissions persuasively. However, the current clustering protocols consume a big part of their energy resources in data-similarity as these protocols periodically sense and forward the same information. In this paper, we introduce a novel two-level Redundant Transmission Control (RTC) approach that ensures the data-similarity using some statistical tests with an appropriate degree of confidence. Later, the Cluster Head (CH) and the Region Head (RH) remove the data-similarity from the original data before forwarding it to the next level. We also introduce a new spatiotemporal and dynamic CH role rotation technique which is capable to adjust the drifted field nodes because of water current movements. The beauty of the proposed model is that the RH controls the communications and redundant transmission between the CH and Mobile Sink (MS), while the CH controls the redundant inner-network transmissions and data-similarity between the cluster members. We conduct simulations to evaluate the performance of our designed framework under different criteria such as average end-to-end delay, the packet delivery ratio, and energy consumption of the network with respect to the recent schemes. The presented results reveal that the proposed model outperforms the current approaches in terms of the selected metrics.

show abstract

“…Disappointingly, Internet-related protocols are not really good at this aspect: The "link rot" problem [44,45] often causes inaccessibility, and target host-centric requests result in repeated transmissions and significant waste on bandwidth [46]. Though remedial measures such as content distribution network (CDN) and peer-to-peer (P2P) overlays have been proposed, these patching methods inevitably introduce extra complexity and unexpected problems, as the high financial cost for content publishers and maintenance cost for service providers in CDN [47]; and a massive amount of cross-ISP traffic which is contradictory to policies of ISPs in P2P [48].…”

Section: Named Content Retrievalmentioning

confidence: 99%

Geo-DMP: A DTN-Based Mobile Prototype for Geospatial Data Retrieval

Xiang

et al. 2019

IJGI

View full text Add to dashboard Cite

Geospatial information is gaining immense interest and importance as we enter the era of highly developed transportation and communication. Despite the proliferation of cellular network and WiFi, on some occasions, users still face barriers to accessing geospatial data. In this paper, we design and implement a distributed prototype system with a delay/disruption tolerant network (DTN), named Geo-DMP, for cooperatively and opportunistically sharing and exchanging named geospatial contents in a device-to-device fashion. First of all, we construct a lightweight “content agent” module to bridge the gap between the application layer and the underlying DTN protocol stack. Afterwards, to profile the mobility history of users in practical geospatial environments, we present a map segmentation scheme based on road network and administrative subdivision information. Subsequently, we associate the regional movement history information with the content retrieval process to devise a hierarchical and region-oriented DTN routing scheme for both requests and responses. Finally, we conduct extensive experiments with real-world trajectories and complete implementations on the emulation platform composed of virtual machines. The experiments corroborate that Geo-DMP has the capability of successfully retrieving geospatial contents for users for most of the time under mobile circumstances with episodic connectivity. Moreover, en-route caches can be efficiently exploited to provision contents from multiple sources with less network resource consumption and shorter user-perceived latencies.

show abstract

On Protocol-Independent Data Redundancy Elimination

Cited by 25 publications

References 37 publications

Scalable and Multifaceted Search and Its Application for Binary Malware Files

Scalable and Multifaceted Search and Its Application for Binary Malware Files

An Energy-Efficient Redundant Transmission Control Clustering Approach for Underwater Acoustic Networks

Geo-DMP: A DTN-Based Mobile Prototype for Geospatial Data Retrieval

Contact Info

Product

Resources

About