An Observational Investigation of Reverse Engineers' Process and Mental Models

Abstract

Reverse engineering is a complex process essential to software-security tasks such as vulnerability discovery and malware analysis. Significant research and engineering effort has gone into developing tools to support reverse engineers. However, little work has been done to understand the way reverse engineers think when analyzing programs, leaving tool developers to make interface design decisions based only on intuition.This paper takes a first step toward a better understanding of reverse engineers' processes, with the goal of producing insights for improving interaction design for reverse engineering tools. We present the results of a semi-structured, observational interview study of reverse engineers (N=16). Each observation investigated the questions reverse engineers ask as they probe a program, how they answer these questions, and the decisions they make throughout the reverse engineering process. From the interview responses, we distill a model of the reverse engineering process, divided into three phases: overview, sub-component scanning, and focused experimentation. Each analysis phase's results feed the next as reverse engineers' mental representations become more concrete. We find that reverse engineers typically use static methods in the first two phases, but dynamic methods in the final phase, with experience playing large, but varying, roles in each phase. Based on these results, we provide five interaction design guidelines for reverse engineering tools.

Challenges and Opportunities in Collaborative Vulnerability Research Workflows

“…This paper presented a coarse-grained, general workflow model of vulnerability research practices. The structure and content of our model aligns and expands upon with the model presented in Votipka et al (2019), suggesting that these findings will be replicable in further studies with larger participant pools, and will provide an empirical grounding for related research in VR. We would like to call particular attention to the need for further research and studies in the following areas.…”

“…This finding spurred additional research to integrate CRSs with human vulnerability researchers in a collaborative environment (Fraze, 2017;Shoshitaishvili et al, 2017). However, there is a very limited information in the literature that empirically examines VR work habits, behaviors, and practices, excepting the work of Votipka, Rabin, Micinski, Foster, & Mazurek (2019), limiting designers' and engineers' ability to effectively integrate CRSs into a collaborative humanmachine system.…”

Effects of Precise and Imprecise Value-Set Analysis (VSA) Information on Manual Code Analysis

“…Human studies in this domain have used a range of techniques to gain process insights, including post-hoc analysis or surveys as in [9] (potentially missing automatic decisions), think-aloud protocols or semi-structured interviews as in [45] (potentially changing analysts' approaches), or in situ data collection (potentially losing information about the relevance of actions). Other studies, like ours, take the general approach used in cognitive science research: developing a task that explores the cognitive processes of interest but is controlled enough to support understanding the data.…”

Declarative Demand-Driven Reverse Engineering

“…While RE tasks are often partially automated (e.g., via decompilation), full automation is often impossible: the extreme semantic expressivity afforded to binaries (including encrypted code, stripped symbol tables, etc..) often necessitates open-ended exploration and case-specific reasoning. Recent literature suggests that many practicioners follow an iterative approach involving several rounds of hypothesis formation and validation/falsification, often assisated via a combination of static and dynamic analysis [1]- [3].…”

“…To rapidly interact with a binary, RE practicioners often use reverse engineering tools such as Ghidra [4], IDA Pro [5], or Radare2 [6]. The goal of these tools is to allow an RE 1 to quickly explore the binary and visualize it (typically D 3 RE vision. d3re allows REs to interactively define and calculate queries of arbitrary complexity over large production binaries and then visualize their results using Ghidra.…”