Using NLP techniques for file fragment classification

A New Approach to Compressed File Fragment Identification

“…Due to compression algorithms, statistical properties of data cannot be used to classify the deflate-encoded data from different file formats. This fact is the reason why previous approaches that exploit statistical properties of compressed data as feature vectors brought low accurate rate [9,18]. Even from the empirical approach of [1], the authors took the advantage of compression properties such as Huffman table size, the detection rate is still low.…”

“…Therefore, they have been included in the data set for many research works. Most of current approaches provide low identification rates to file fragments of compound files which are less than 30 % as reported in [8,9]. Other works bring better identification rates but with much smaller size or small number of file types as considered in [10,11].…”

“…Thirdly, SVM with byte frequency distribution as feature vector can be used to efficiently identify these inflate data. Moreover, it is demonstrated that SVM can provide high accuracy and performance to recognize the data fragments which have low or medium entropy values [9,11]. In addition, it is showed that entropy-based clustering before feeding data into machine learning techniques might bring higher accuracy [15].…”

Audio Fragment Identification System

“…The statistical patterns denote quantitative features such as mean, variance and frequency, whereas the structural patterns denote morphological features such as syntactic grammar and interrelationship [21]. Recently, many researches of format classification involve statistical approaches as in [15][16][17][18]. Because of the rapid growth of the capacity of multimedia, many formats utilize compression methods to reduce the cost and lead to generate high entropy data.…”

“…SVM is a powerful machine learning method because it is not limited by number of samples and dimensionality [11][12][13][14]. In [15][16][17][18], researchers used statistical features, such as mean, standard deviation, byte frequency distribution, Shannon entropy, N-gram and Hamming weight to classify the formats. Because of strong compression and entropy coding of audio file, it is hard to achieve high accuracy of classification only with statistical features.…”