Polymorphic type inference for machine code

Abstract

For many compiled languages, source-level types are erased very early in the compilation process. As a result, further compiler passes may convert type-safe source into type-unsafe machine code. Type-unsafe idioms in the original source and type-unsafe optimizations mean that type information in a stripped binary is essentially nonexistent. The problem of recovering high-level types by performing type inference over stripped machine code is called type reconstruction, and offers a useful capability in support of reverse engineering and decompilation.In this paper, we motivate and develop a novel type system and algorithm for machine-code type inference. The features of this type system were developed by surveying a wide collection of common source-and machine-code idioms, building a catalog of challenging cases for type reconstruction. We found that these idioms place a sophisticated set of requirements on the type system, inducing features such as recursively-constrained polymorphic types. Many of the features we identify are often seen only in expressive and powerful type systems used by high-level functional languages.Using these type-system features as a guideline, we have developed Retypd: a novel static type-inference algorithm for machine code that supports recursive types, polymorphism, and subtyping. Retypd yields more accurate inferred types than existing algorithms, while also enabling new capabilities * such as reconstruction of pointer const annotations with 98% recall. Retypd can operate on weaker program representations than the current state of the art, removing the need for highquality points-to information that may be impractical to compute.

Polymorphic type inference for machine code

“…Let us briefly see how operations in the original program translate to type constraints, using C-like pseudocode for clarity. The full conversion from disassembly to type constraints is described in Appendix A of [22]. Value copies: When a value is moved between program variables in an assignment like x := y, we make the conservative assumption that the type of x may be upcast to a supertype of y.…”

“…But we still maintain this capability at procedure invocations due to our use of polymorphic type schemes. An explanation of how type-scheme instantiation enables us to forget fields of an object appears in §3.4, with more details in §E.1.2 of [22].…”

“…The deduction rules of Figure 3 are simple enough that each proof may be reduced to a normal form (see Theorem B.1 in [22]). An encoding of the normal forms as transition sequences in a modified pushdown system is used to provide a compact representation of the entailment closure C = {c | C � c}.…”

“…By collapsing isomorphic subtrees, we can represent sketches as deterministic finite state automata with each state labeled by an element of Λ. The set of sketches admits a lattice structure, with operations described by Figure 18 in [22].…”

“…Due to space limitations, details of the proofs and algorithms appear in the appendices, which are available in the online version of this paper at arXiv:1603.05495 [22]. Scripts and data sets used for evaluation also appear there.…”