Reliable detection of compressed and encrypted data

Gaspari, Fabio De; Hitaj, Dorjan; Pagnotta, Giulio; Carli, Lorenzo De; Mancini, Luigi V.

doi:10.1007/s00521-022-07586-7

Cited by 8 publications

(11 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One suggested approach is to further encode the encrypted files in a way as to reduce their overall entropy value, by for example, using base64 encoding. However, despite this theoretical mitigation technique, crypto-ransomware techniques are still being proposed that utilise entropy calculations as part of their design [ 21 , 75 , 76 , 77 , 78 ].…”

Section: Discussionmentioning

confidence: 99%

“…The NIST SP800-22 specification [ 37 , 40 , 45 ] from 2010 describes a suite of tests whose intended use is to evaluate the quality of random number generators [ 21 ]. The suite consists of 15 distinct tests, and which analyse various structural aspects of a byte sequence.…”

Section: Methodsmentioning

confidence: 99%

“…This difference in entropy can be used as an indicator to identify when encrypted files, or more specifically high entropy files, are being written to disk. Significant research has been performed into crypto-ransomware detection using the Shannon entropy calculation [ 6 , 13 , 14 , 15 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 ], resulting in many interesting detection techniques and tools. However, some researchers do comment on the unsuitability of this technique when analysing typically higher entropy files [ 7 , 30 ] such as with archive and compressed files.…”

Section: Related Workmentioning

confidence: 99%

“…There are 2 20 possible 20-letter words. For a truly random string of 2 21 + 19 bits, the number of missing words j should be normally distributed with mean 141,909 and sigma 428. Test 5-Overlapping Pairs Sparse Occupancy (OPSO).…”

Section: Informed Consent Statement: Not Applicablementioning

confidence: 99%

“…Each letter is determined by a specified ten bits from a 32-bit integer in the sequence to be tested. OPSO generates 2 21 (overlapping) 2-letter words and counts the number of missing words that do not appear in the entire sequence [42]. Test 6-Overlapping Quadruples Sparse Occupancy (OQSO) Test.…”

Section: Informed Consent Statement: Not Applicablementioning

confidence: 99%

See 4 more Smart Citations

Comparison of Entropy Calculation Methods for Ransomware Encrypted File Identification

Davies

Macfarlane

Buchanan

2022

Entropy

View full text Add to dashboard Cite

Ransomware is a malicious class of software that utilises encryption to implement an attack on system availability. The target’s data remains encrypted and is held captive by the attacker until a ransom demand is met. A common approach used by many crypto-ransomware detection techniques is to monitor file system activity and attempt to identify encrypted files being written to disk, often using a file’s entropy as an indicator of encryption. However, often in the description of these techniques, little or no discussion is made as to why a particular entropy calculation technique is selected or any justification given as to why one technique is selected over the alternatives. The Shannon method of entropy calculation is the most commonly-used technique when it comes to file encryption identification in crypto-ransomware detection techniques. Overall, correctly encrypted data should be indistinguishable from random data, so apart from the standard mathematical entropy calculations such as Chi-Square (χ2), Shannon Entropy and Serial Correlation, the test suites used to validate the output from pseudo-random number generators would also be suited to perform this analysis. The hypothesis being that there is a fundamental difference between different entropy methods and that the best methods may be used to better detect ransomware encrypted files. The paper compares the accuracy of 53 distinct tests in being able to differentiate between encrypted data and other file types. The testing is broken down into two phases, the first phase is used to identify potential candidate tests, and a second phase where these candidates are thoroughly evaluated. To ensure that the tests were sufficiently robust, the NapierOne dataset is used. This dataset contains thousands of examples of the most commonly used file types, as well as examples of files that have been encrypted by crypto-ransomware. During the second phase of testing, 11 candidate entropy calculation techniques were tested against more than 270,000 individual files—resulting in nearly three million separate calculations. The overall accuracy of each of the individual test’s ability to differentiate between files encrypted using crypto-ransomware and other file types is then evaluated and each test is compared using this metric in an attempt to identify the entropy method most suited for encrypted file identification. An investigation was also undertaken to determine if a hybrid approach, where the results of multiple tests are combined, to discover if an improvement in accuracy could be achieved.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Informed Consent Statement: Not Applicablementioning

confidence: 99%

Section: Informed Consent Statement: Not Applicablementioning

confidence: 99%

See 3 more Smart Citations

Comparison of Entropy Calculation Methods for Ransomware Encrypted File Identification

Davies

Macfarlane

Buchanan

2022

Entropy

View full text Add to dashboard Cite

show abstract

MaleficNet: Hiding Malware into Deep Neural Networks Using Spread-Spectrum Channel Coding

Hitaj

Pagnotta

Hitaj

et al. 2022

Computer Security – ESORICS 2022

Self Cite

View full text Add to dashboard Cite

Training high-quality deep learning models is a challenging task due to computational and technical requirements. A growing number of individuals, institutions, and companies increasingly rely on pre-trained, third-party models made available in public repositories. These models are often used directly or integrated in product pipelines with no particular precautions, since they are effectively just data in tensor form and considered safe.In this paper, we raise awareness of a new machine learning supply chain threat targeting neural networks. We introduce MaleficNet 2.0, a novel technique to embed self-extracting, selfexecuting malware in neural networks. MaleficNet 2.0 uses spread-spectrum channel coding combined with error correction techniques to inject malicious payloads in the parameters of deep neural networks. MaleficNet 2.0 injection technique is stealthy, does not degrade the performance of the model, and is robust against removal techniques. We design our approach to work both in traditional and distributed learning settings such as Federated Learning, and demonstrate that it is effective even when a reduced number of bits is used for the model parameters. Finally, we implement a proof-of-concept self-extracting neural network malware using MaleficNet 2.0, demonstrating the practicality of the attack against a widely adopted machine learning framework.Our aim with this work is to raise awareness against these new, dangerous attacks both in the research community and industry, and we hope to encourage further research in mitigation techniques against such threats.

show abstract

Evading behavioral classifiers: a comprehensive analysis on evading ransomware detection techniques

Gaspari

Hitaj

Pagnotta

et al. 2022

Neural Comput & Applic

Self Cite

View full text Add to dashboard Cite

Recent progress in machine learning has led to promising results in behavioral malware detection. Behavioral modeling identifies malicious processes via features derived by their runtime behavior. Behavioral features hold great promise as they are intrinsically related to the functioning of each malware, and are therefore considered difficult to evade. Indeed, while a significant amount of results exists on evasion of static malware features, evasion of dynamic features has seen limited work. This paper examines the robustness of behavioral ransomware detectors to evasion and proposes multiple novel techniques to evade them. Ransomware behavior differs significantly from that of benign processes, making it an ideal best case for behavioral detectors, and a difficult candidate for evasion. We identify and propose a set of novel attacks that distribute the overall malware workload across a small set of independent, cooperating processes in order to avoid the generation of significant behavioral features. Our most effective attack decreases the accuracy of a state-of-the-art classifier from 98.6 to 0% using only 18 cooperating processes. Furthermore, we show our attacks to be effective against commercial ransomware detectors in a black-box setting. Finally, we evaluate a detector designed to identify our most effective attack, as well as discuss potential directions to mitigate our most advanced attack.

show abstract

Reliable detection of compressed and encrypted data

Cited by 8 publications

References 29 publications

Comparison of Entropy Calculation Methods for Ransomware Encrypted File Identification

Comparison of Entropy Calculation Methods for Ransomware Encrypted File Identification

MaleficNet: Hiding Malware into Deep Neural Networks Using Spread-Spectrum Channel Coding

Evading behavioral classifiers: a comprehensive analysis on evading ransomware detection techniques

Contact Info

Product

Resources

About