Survey of Protocol Reverse Engineering Algorithms: Decomposition of Tools for Static Traffic Analysis

Abstract

Knowledge about a network protocol to understand the communication between entities is necessary for vulnerability research, penetration testing, malware analysis, network reconnaissance, and network modeling. Traffic analysis is one approach to infer a protocol, and this approach has specific challenges, tasks, methods, and solutions. In this survey, we collect tools presented by prior research in the field of protocol reverse engineering by static traffic trace analysis. We dissect each tool to discern the individual mechanisms and the algorithms on which they are based, then categorize and contrast the mechanisms and algorithms used in static traffic trace analysis to discuss how successfully they were applied in each case. To structure our discussion about the tools, we compared classification schemes for protocol reverse engineering. We present and discuss an explicit process model for static traffic trace analysis to reveal the common structure of the decomposed tools and frameworks from previous research. Via discussions of the algorithms applied within each tool, we show relations between tools, methods, and the process for each process task. We validate our model by applying it to each of the tools, then provide an outline of the utility of protocol reverse engineering. Beginning with the process description, we deduce which solutions and algorithms have already been investigated and where challenges remain to determine how new solutions may be researched in the future. Across the entire field of protocol reverse engineering, few implementations of tools and frameworks are publicly available, which remains a prevalent problem.

[Retracted] Dynamic Analysis of Network Protocol Hiding Behavior Based on Perceptual Mining

“…For example, when the program does not output data, the writing data function would not be easy to obtain. Kleber et al [7] proposed a scheme to mine the message format based on the data change distribution characteristics of the message. e theoretical basis of the scheme is to statistically analyze the change characteristics of the value range of each field in the message.…”

Message Type Identification of Binary Network Protocols using Continuous Segment Similarity

“…The use of sequence alignment to perform such an analysis was first suggested by Beddoe [16]. Since Beddoe's paper, a variety of algorithms from natural language processing and bio-informatics have been applied to network protocols [3]. There are also practical implementations to perform static traffic analysis, the most versatile of which is Netzob [10].…”

“…Several recent surveys by Narayan et al [1], Duchêne et al [2], and Kleber et al [3] describe the current state of the art for protocol reverse engineering based on network traffic traces or programs. In this paper, we focus on traffic analysis based on information gained by observing only the communication link.…”

Unsupervised Binary Protocol Clustering Based on Maximum Sequential Patterns

“…The PRE process of the protocol includes input preprocessing, protocol format extraction and state machine inference [20][21][22]. First, the input preprocessing captures protocol messages on the network and transforms them into a format suitable for subsequent analysis.…”