Analyzing Network Protocols of Application Layer Using Hidden Semi-Markov Model

Abstract

With the rapid development of Internet, especially the mobile Internet, the new applications or network attacks emerge in a high rate in recent years. More and more traffic becomes unknown due to the lack of protocol specifications about the newly emerging applications. Automatic protocol reverse engineering is a promising solution for understanding this unknown traffic and recovering its protocol specification. One challenge of protocol reverse engineering is to determine the length of protocol keywords and message fields. Existing algorithms are designed to select the longest substrings as protocol keywords, which is an empirical way to decide the length of protocol keywords. In this paper, we propose a novel approach to determine the optimal length of protocol keywords and recover message formats of Internet protocols by maximizing the likelihood probability of message segmentation and keyword selection. A hidden semi-Markov model is presented to model the protocol message format. An affinity propagation mechanism based clustering technique is introduced to determine the message type. The proposed method is applied to identify network traffic and compare the results with existing algorithm.

Survey of Protocol Reverse Engineering Algorithms: Decomposition of Tools for Static Traffic Analysis

“…They deduced design decisions for fields in these protocols to obtain hypotheses on field sizes and behavior. 9) Cai et al (2016): An idea similar to Whalen et al [19] for extracting message formats was proposed by Cai et al [25], relying completely on Markov Models. In contrast to Whalen et al, the new approach requires no previous knowledge of field boundary candidates.…”

“…At the end of each section, we conclude the application of the presented methods and algorithms for the respective [11] RolePlayer [18] Discoverer [13] Whalen et al [19] Biprominer [20] ProDecoder [21] Li et al [22] FieldHunter [23,24] Cai et al [25] PRE-Bin [26] Xiao et al [27] NEMESYS [28] ScriptGen [29] PEXT [30] Trifilo et al [31] Veritas [32] PREUGI [33] AutoFuzz [34] ReverX [14] ASAP [35] PRISMA [15] AutoReEngine [17] Netzob [16] process step. This discussion is intended to provide lessons learned from the application of the methods.…”

A Survey of Automatic Protocol Reverse Engineering Approaches, Methods, and Tools on the Inputs and Outputs View

“…This section analyzes 39 different approaches, methods, and tools outputs' focus in relation to the algorithms and/or technique along with architectures applied. [4] 2007 I PFSM for future work Polyglot [5] 2007 I Dispatcher [6] 1st work AutoFormat [7] 2008 I ∼ Tupni [8] 2008 I PFSM for future work ReFormat [9] 2009 I Decryption before PRE Prospex [10] 2009 I Reverse engineers PFSM as well ProDecoder [11] 2012 I ∼ Wang et al [12] 2013 I PRE in wireless environment ProGraph [13] 2015 I Traffic classification Cai et al [14] 2016 In the Tokenization and Initial Clustering module, Discoverer clusters messages based on their token patterns assigned in 4-tuple information (dir, class of token 1, class of token 2,. . ., class of token n) where dir is the direction of the message C2S or S2C followed by the classes of all tokens in a given message.…”

“…Cai et al [14] aim at selecting fields keywords with their optimal lengths and finding about how protocol messages are segmented. Through a hidden semi-Markov model Cai et al try to model an entire protocol message format.…”

“…Analyzed/reverse engineered protocols Related approach, method, or tool 7-application HTTP, HTTPS, DNS, SIP, FTP, TFTP, IMAP, SSH, IRC, XMPP, SMTP, BitTorrent, MIME, SNMP, MSNP, POP3, DHCP, SMB, CIFS, Samba, XunLei, BitSpirit, QQLive, eMule, ICQ, SopCast, PPLIVE, PPTV, KAD, ISAKMP, SSDP, WeChat, Kugoo [3-11, 13, 14, 16-22, 24-26, 28-30, 32-37, 39, 41, 42] 6-presentation SMB, WMF, BMP, JPG, PNG, TIF, MIME, SSL [8-11, 29, 39] 5-session RPC, NetBIOS, NFS, SSL [8,14,29,30,39,40] 4-transport TCP, UDP, SrvLoc [17,23,31,38] 3-network ICMP, OSPF, RIP [7,12,40] 2-data link ARP, STP [12,15,17,23] 1-physical none [15] analyzed or reverse engineered by 33 out of 39 approaches, methods, and tools presented in this paper. HTTP protocol appears in 21 approaches as the most analyzed protocol.…”