Comparison of Entropy Calculation Methods for Ransomware Encrypted File Identification

Abstract

Ransomware is a malicious class of software that utilises encryption to implement an attack on system availability. The target’s data remains encrypted and is held captive by the attacker until a ransom demand is met. A common approach used by many crypto-ransomware detection techniques is to monitor file system activity and attempt to identify encrypted files being written to disk, often using a file’s entropy as an indicator of encryption. However, often in the description of these techniques, little or no discussion is made as to why a particular entropy calculation technique is selected or any justification given as to why one technique is selected over the alternatives. The Shannon method of entropy calculation is the most commonly-used technique when it comes to file encryption identification in crypto-ransomware detection techniques. Overall, correctly encrypted data should be indistinguishable from random data, so apart from the standard mathematical entropy calculations such as Chi-Square (χ2), Shannon Entropy and Serial Correlation, the test suites used to validate the output from pseudo-random number generators would also be suited to perform this analysis. The hypothesis being that there is a fundamental difference between different entropy methods and that the best methods may be used to better detect ransomware encrypted files. The paper compares the accuracy of 53 distinct tests in being able to differentiate between encrypted data and other file types. The testing is broken down into two phases, the first phase is used to identify potential candidate tests, and a second phase where these candidates are thoroughly evaluated. To ensure that the tests were sufficiently robust, the NapierOne dataset is used. This dataset contains thousands of examples of the most commonly used file types, as well as examples of files that have been encrypted by crypto-ransomware. During the second phase of testing, 11 candidate entropy calculation techniques were tested against more than 270,000 individual files—resulting in nearly three million separate calculations. The overall accuracy of each of the individual test’s ability to differentiate between files encrypted using crypto-ransomware and other file types is then evaluated and each test is compared using this metric in an attempt to identify the entropy method most suited for encrypted file identification. An investigation was also undertaken to determine if a hybrid approach, where the results of multiple tests are combined, to discover if an improvement in accuracy could be achieved.

Entropy Sharing in Ransomware: Bypassing Entropy-Based Detection of Cryptographic Operations

“…Therefore, an increase in entropy can be considered an important metric for ransomware detection. However, here we note that just relying on the calculated Shannon entropy value to distinguish between encrypted and non-encrypted files would be a difficult task generating a lot of false positives and negatives [ 39 ]. In Section 5 , we show our experimental result on this issue.…”

A Study on Countermeasures against Neutralizing Technology: Encoding Algorithm-Based Ransomware Detection Methods Using Machine Learning

“…In other words, information entropy refers to the uniformity of data, and if the data are uniform, they have high entropy. Conversely, if the data are not uniform and are deviated, they have relatively low entropy [8][9][10]. In this case, entropy can range from zero to eight.…”

“…Many ransomware programs have the feature of maintaining the file type representing the information of the original file in the file name to seamlessly decode the encrypted file when the user pays for it [8]. Due to these characteristics, even if a file is infected with ransomware, the file type can be determined; therefore, it is believed that it is possible to learn a model using the file type characteristics and detect a file infected with ransomware.…”

“…file when the user pays for it [8]. Due to these characteristics, even if a file is ransomware, the file type can be determined; therefore, it is believed that it learn a model using the file type characteristics and detect a file infected with Therefore, to confirm whether a given file type can be exploited as a feature, infection with the actual WannaCry ransomware for xls, pdf, and doc files Figure 4.…”

Neutralization Method of Ransomware Detection Technology Using Format Preserving Encryption

“…As a related work, [ 20 ] proposed a zone division-based ransomware detection method to detect ransomware by separating areas containing file metadata, such as file headers, footers, signature information, and file contents that define file extensions in binary data. In [ 15 ], features for entropy distribution were derived for various file formats and machine learning models, such as KNN (K-Nearest Neighbor), linear regression, ridge regression, logistics regression, decision tree, random forest, SVM (Support Vector Machine), and MLP (Multi-Layer Perception), which were applied.…”