Fast Preprocessing by Suffix Arrays for Managing Byte n-grams to Detect Malware Subspecies by Machine Learning

Abstract

Although machine learning methods with byte n-grams have been marking high score for classifying malware and benignware, they seem not to be used for current anti-virus software. A performance bottleneck of the methods is dealing with byte n-grams in preprocessing such as top-k selection. It takes a long time to extract all byte n-grams which are required for selecting top-k n-grams. Moreover, if several "n"s are wanted to be used such as 4grams, 8-grams and 16-grams, n-grams with each "n" must be extracted again and again. Therefore, we proposed a fast preprocessing method of extracting n-grams by applying a suffix array algorithm. Furthermore, our method can manage multi-length byte n-grams at the same time. In addition, selecting feature n-grams like top-k n-grams with information gain is also included in our method. On the other hand, our method has a limitation that it is only applicable to a large number of samples in the same malware subspecies family, which become extinct. We evaluated the speed of our method by comparing with usual ways. We also evaluated our method by machine learning with actual samples in four old malware subspecies families. We think there is a hope that our method may be applicable to detecting current targeted malware.