Approaches to the classification of high entropy file fragments

Penrose, Philip; Macfarlane, Richard; Buchanan, William J.

doi:10.1016/j.diin.2013.08.004

Cited by 29 publications

(21 citation statements)

References 25 publications

(37 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This task is a specialization of the problem of file type classification, well-known in computer forensics. In this context, many statistical techniques have been proposed, usually leveraging differences in the distribution of byte frequency among different file types [32,33,34,35]. Forensics techniques usually aim to classify all executable files in the same class, thus are not applicable as-is to our problem.…”

Section: Related Workmentioning

confidence: 99%

ELISA: ELiciting ISA of Raw Binaries for Fine-Grained Code and Data Separation

Nicolao

Pogliani

Polino

et al. 2018

Detection of Intrusions and Malware, and Vulnerability Assessment

View full text Add to dashboard Cite

Static binary analysis techniques are widely used to reconstruct the behavior and discover vulnerabilities in software when source code is not available. To avoid errors due to mis-interpreting data as machine instructions (or vice-versa), disassemblers and static analysis tools must precisely infer the boundaries between code and data. However, this information is often not readily available. Worse, compilers may embed small chunks of data inside the code section. Most state of the art approaches to separate code and data are rooted on recursive traversal disassembly, with severe limitations when dealing with indirect control instructions. We propose ELISA, a technique to separate code from data and ease the static analysis of executable files. ELISA leverages supervised sequential learning techniques to locate the code section(s) boundaries of header-less binary files, and to predict the instruction boundaries inside the identified code section. As a preliminary step, if the Instruction Set Architecture (ISA) of the binary is unknown, ELISA leverages a logistic regression model to identify the correct ISA from the file content. We provide a comprehensive evaluation on a dataset of executables compiled for different ISAs, and we show that our method is capable to identify code sections with a byte-level accuracy (F1 score) ranging from 98.13% to over 99.9% depending on the ISA. Fine-grained separation of code from embedded data on x86, x86-64 and ARM executables is accomplished with an accuracy of over 99.9%.

show abstract

Section: Related Workmentioning

confidence: 99%

ELISA: ELiciting ISA of Raw Binaries for Fine-Grained Code and Data Separation

Nicolao

Pogliani

Polino

et al. 2018

Detection of Intrusions and Malware, and Vulnerability Assessment

View full text Add to dashboard Cite

show abstract

“…One part of the data set contains approximately one million files collected using random searches of the .gov domain. It has been used by many ransomware researchers [14,27,37,44,46,47,48,50,51,52,55], supporting the claim that this data set is a well-known and respected source of test data. In 2017 Grajeda et al [37] reported that this was the most popular data set currently in use.…”

Section: Govdocs1mentioning

confidence: 90%

“…Actions performed to ensure that the data contained within the data set is of the highest quality are also described. The breadth and depth of the NapierOne data set should also remove the need for other researchers to complement the data set with additional files as has previously been the case [47,48,51].…”

Section: Discussionmentioning

confidence: 99%

NapierOne: A modern mixed file data set alternative to Govdocs1

Davies,

Macfarlane,

Buchanan

2022

Preprint

Self Cite

View full text Add to dashboard Cite

It was found when reviewing the ransomware detection research literature that almost no proposal provided enough detail on how the test data set was created, or sufficient description of its actual content, to allow it to be recreated by other researchers interested in reconstructing their environment and validating the research results. A modern cybersecurity mixed file data set called NapierOne is presented, primarily aimed at, but not limited to, ransomware detection and forensic analysis research. NapierOne was designed to address this deficiency in reproducibility and improve consistency by facilitating research replication and repeatability. The methodology used in the creation of this data set is also described in detail. The data set was inspired by the Govdocs1data set and it is intended that NapierOne be used as a complement to this original data set.An investigation was performed with the goal of determining the common files types currently in use. No specific research was found that explicitly provided this information, so an alternative consensus approach was employed. This involved combining the findings from multiple sources of file type usage into an overall ranked list. After which 5,000 real-world example files were gathered, and a specific data subset created, for each of the common file types identified. In some circumstances, multiple data subsets were created for a specific file type, each subset representing a specific characteristic for that file type. For example, there are multiple data subsets for the ZIP file type with each subset containing examples of a specific compression method. Ransomware execution tends to produce files that have high entropy, so examples of file types that naturally have this attribute are also present. The resulting entire data set comprises of a 100 separate data subsets divided between 44 distinct file types, resulting in almost 500,000 unique files in total. A description of the techniques used to gather the files for each file type is provided together with the actions that were performed on the files to confirm that they were of the highest quality and provided an accurate representation of their specific file type. Details are also provided on the content of the entire data set as well as instructions on how researchers can gain free and unlimited access to the final data set.While the data set was initially created to aid research in ransomware detection, it is sufficiently broad and diverse enough to allow for its application in many other areas of research that require a varied mixture of common real-world file examples. The NapierOne data set is an ongoing project and researchers are strongly encouraged to leverage this data set in their own research.

show abstract

“…Some data fragments are quite easy to identify their file types, such as fragments that belong to files having only unified data such as ASCII text. Conversely, there is no powerful method to identify fragments that are data portions of high entropy files [3,4]. High entropy files normally contain compressed data, such as zip archives.…”

Section: Introductionmentioning

confidence: 99%

“…File fragment identification is one of the most serious problems in the process of file carving applied for digital forensics [1][2][3]. Some data fragments are quite easy to identify their file types, such as fragments that belong to files having only unified data such as ASCII text.…”

Section: Introductionmentioning

confidence: 99%

A New Approach to Compressed File Fragment Identification

Nguyen

Tran

et al. 2015

Advances in Intelligent Systems and Computing

View full text Add to dashboard Cite

Identifying the underlying type of a file given only a file fragment is a big challenge in digital forensics. Many methods have been applied to file type identification; however the identification accuracies of most of file types are still very low, especially for files having complex structures because their contents are compound data built from different data types. In this paper, we propose a new approach based on the deflate-encoded data detection, entropy-based clustering, and the use of machine learning techniques to identify deflate-encoded file fragments. Experiments on the popular compound file type showed high identification accuracy for the proposed method.

show abstract

Approaches to the classification of high entropy file fragments

Cited by 29 publications

References 25 publications

ELISA: ELiciting ISA of Raw Binaries for Fine-Grained Code and Data Separation

ELISA: ELiciting ISA of Raw Binaries for Fine-Grained Code and Data Separation

NapierOne: A modern mixed file data set alternative to Govdocs1

A New Approach to Compressed File Fragment Identification

Contact Info

Product

Resources

About