Application of sim-hash algorithm and big data analysis in spam email detection system

Ho, Phuc-Tran; Kim, Hee-Sun; Kim, Sung-Ryul

doi:10.1145/2663761.2664221

Cited by 9 publications

(2 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Simhashing has wide-ranging applications from detecting duplicates in texts (e.g. websites) to different security and to malware analysis, specifically with the Hamming distance similarity measure [25,26,27]. Inspired from NLP application domain, a n-gram is a contiguous sequence of n items (here, a byte pair) from a given sequence of the binary file.…”

Section: The Feature Extractionmentioning

confidence: 99%

Malytics: A Malware Detection Scheme

et al. 2018

View full text Add to dashboard Cite

An important problem of cyber-security is malware analysis. Besides good precision and recognition rate, a malware detection scheme needs to be able to generalize well for novel malware families (a.k.a zero-day attacks). It is important that the system does not require excessive computation particularly for deployment on the mobile devices.In this paper, we propose a novel scheme to detect malware which we call Malytics. It is not dependent on any particular tool or operating system. It extracts static features of any given binary file to distinguish malware from benign. Malytics consists of three stages: feature extraction, similarity measurement and classification. The three phases are implemented by a neural network with two hidden layers and an output layer. We show feature extraction, which is performed by tf -simhashing, is equivalent to the first layer of a particular neural network. We evaluate Malytics performance on both Android and Windows platforms. Malytics outperforms a wide range of learning-based techniques and also individual state-of-the-art models on both platforms. We also show Malytics is resilient and robust in addressing zero-day malware samples. The F1-score of Malytics is 97.21% and 99.45% on Android dex file and Windows PE files respectively, in the applied datasets. The speed and efficiency of Malytics are also evaluated.

show abstract

Section: The Feature Extractionmentioning

confidence: 99%

Malytics: A Malware Detection Scheme

et al. 2018

View full text Add to dashboard Cite

show abstract

“…While min-hash uses many hash values to represent a document, having each value computed with a different hash function, simhash gives a more compact output by reducing document vectors to a small sized real-valued fingerprints (Charikar, 2002). Simhash was successfully evaluated for duplicate detection of web pages (Manku et al, 2007;Henzinger, 2006), code segments (Uddin et al, 2011), short messages (Pi et al, 2009), spam (Ho et al, 2014) and academic papers (Williams and Giles, 2013). Our contribution to the literature is in the use of simhash fingerprinting for larger texts in form of digital books.…”

Section: Related Workmentioning

confidence: 99%