Similarity Digest Search: A Survey and Comparative Analysis of Strategies to Perform Known File Filtering Using Approximate Matching

Abstract

Digital forensics is a branch of Computer Science aiming at investigating and analyzing electronic devices in the search for crime evidence. There are several ways to perform this search. Known File Filter (KFF) is one of them, where a list of interest objects is used to reduce/separate data for analysis. Holding a database of hashes of such objects, the examiner performs lookups for matches against the target device. However, due to limitations over hash functions (inability to detect similar objects), new methods have been designed, called approximate matching. This sort of function has interesting characteristics for KFF investigations but suffers mainly from high costs when dealing with huge data sets, as the search is usually done by brute force. To mitigate this problem, strategies have been developed to better perform lookups. In this paper, we present the state of the art of similarity digest search strategies, along with a detailed comparison involving several aspects, as time complexity, memory requirement, and search precision. Our results show that none of the approaches address at least these main aspects. Finally, we discuss future directions and present requirements for a new strategy aiming to fulfill current limitations.

Security Hardened and Privacy Preserved Android Malware Detection Using Fuzzy Hash of Reverse Engineered Source Code

“…In this work, we provide a solution for malware (repacked) detection using static analysis with fuzzy hashes. Two approaches are used for malware analysis, i.e., static analysis [12] and dynamic analysis [13]. In this study, our focus is on the static analysis of malware with the help of fuzzy hashes.…”

Research Trends, Challenges, and Emerging Topics in Digital Forensics: A Review of Reviews

“…In the context of database forensics and data acquisition, the challenges of big data analysis and data mining techniques for digital forensics [72], [73], and text clustering [74] were investigated. Moreover, a survey of techniques to perform similarity digest search is provided in [75].…”

“…[58], [61], [66], [67], [69], [75] Lack of standardized tools and technologies [59], [65], [66], [69], [70], [74] Forensic seizure and analysis of proprietary and/or distributed filesystems [58]- [60], [70], [71], [71], [73] Variety of format and content type. Not standard logging features and settings [61], [65]- [70], [73], [75] No validation/verification in real-life scenarios and large datasets [72], [74] Subjectivity of the evaluation of content retrieval algorithms [72], [74] Advanced knowledge and training of analysts and investigators [69], [72] Lack of guidance for investigators regarding selective search and seize.…”

“…[58], [61], [66], [67], [69], [75] Lack of standardized tools and technologies [59], [65], [66], [69], [70], [74] Forensic seizure and analysis of proprietary and/or distributed filesystems [58]- [60], [70], [71], [71], [73] Variety of format and content type. Not standard logging features and settings [61], [65]- [70], [73], [75] No validation/verification in real-life scenarios and large datasets [72], [74] Subjectivity of the evaluation of content retrieval algorithms [72], [74] Advanced knowledge and training of analysts and investigators [69], [72] Lack of guidance for investigators regarding selective search and seize. Subjectivity of search terms based on investigator's experience [69], [74] Difficulty to apply low-level analysis techniques, hindering correctness of the results [62], [70] Sophisticated malware implementing antiforensic techniques [61]- [63], [74] Volatile data acquisition due to hardware constraints [71] Stealthy non-memory-resident malware [64] Handling, execution and monitoring of memory [61], [63], […”

Avaliação do impacto da remoção de características comuns em estratégias de busca de arquivos similares

“…O método para tornar a busca de similaridade mais eficiente varia para cada estratégia, podendo ser a utilização de tabelas hash ou deárvores de filtros de Bloom [Moia and Henriques 2017]. Apesar das vantagens trazidas por estas estratégias, elas apresentam problemas em relaçãoà alta quantidade de falsos positivos, istoé, os pares de arquivos que as estratégias afirmam serem similares não se provam similares quando se examina o conteúdo do dado, gerado pelo usuário.…”