Packet Inspection for Unauthorized OS Detection in Enterprises

Abstract

Many recent malware implementations employ virtual machines to carry out malicious activities. ese are hard to detect because antivirus software running in the native OS can't detect virtual machines' system states. An approach that uses TCP SYN packets for OS fi ngerprinting can detect the presence of unauthorized OSs.M any modern malware implementations carry out their activities using virtual machines to escape detection from antivirus so ware running on the host OS. 1 is malware can also be used as part of a botnet to transfer information from the infected machine to a command-and-control center. Such malware is hard to detect because the context data and state of the programs run by a virtual machine can't be accessed by antivirus so ware installed on the native OS.Due to increasing OS vulnerabilities, enterprise network administrators must regularly perform OS audits. ese help determine various services running on different systems and identify OSs with aws that might cause vulnerabilities in the enterprise network. Audits also help in con guring network-based intrusion detection systems and maintaining an adaptive enterprise security policy. If OSs' virtual machines di er from the native OSs, these malicious OSs can be identi ed and the infected machines can be cleaned. Although various RFCs specify de nitions and interpretations of different TCP/IP packet elds, 2-7 many fail to specify a standard set of initial values for these elds. As a consequence, developers of various OSs implement the protocol stack with di erent initial values for these elds.Similar to the way a human ngerprint serves as a tool to uniquely identify a person, an OS can be uniquely identi ed on a network by its packet ngerprint. Packet ngerprints are derived from the implementation dissimilarities of various OSs' communication protocols. By analyzing initial values of certain protocol ags, options, packet elds, and data in the packets that a host sends over a network, we can determine the OS installed in a host. If the OS determined from the packet generated by an enterprise host di ers from the original OS installed in the host machine, an unauthorized OS is likely present. An unauthorized OS might have been installed either by a malicious user without obtaining permission from a network administrator or by a virtual machine installed by malware over a native host OS.In this article, we present an approach to passively ngerprint OSs using the information available from TCP packets as well as a system to use this approach to detect unauthorized OSs in an enterprise network. For more information on related work in OS ngerprinting, see the sidebar. Figure 1 shows the outline of our proposed system. e enterprise network connects to the Internet through a rewall that is typically combined with a router. e enterprise router captures outgoing TCP SYN packets from the network, which are forwarded to an OS ngerprinting analyzer that extracts and analyzes their headers. Our OS ngerprinting analyzer applies a Euclidean distance estimation algo...

Trust and Risk Assessment in IoT Networks

“…The combination of TTL and window size fields is used to fingerprint devices using operating system vendor-specific values in their implementations for detection. Table 1 lists some commonly used and legacy operating systems and the window size/TTL pairs used for each [26].…”

A detection method of lost assets based on feature optimization and active-passive detection

“…The device fingerprint is expanded to improve the accuracy of network asset detection and traceability. Tyagi et al 14 proposed an operating system equipment asset detection method based on Euclidean distance, which is used to discover unauthorized abnormal equipment hosts in the system, which shortens the modeling time compared to other complex classifiers. Reference 15 used the C4.5 decision tree model to achieve passive detection of device fingerprints based on TCP/IP protocol stack with a shorter modeling time and higher accuracy, which improves the detection rate of fingerprints that are not accurately matched.…”

Automatic Identification of Honeypot Server Using Machine Learning Techniques

“…is TCP/IP fingerprints can help people recognize different devices, including honeypots. Some common TCP/IP fingerprints are TCP FIN flag, TCP initial window, and ACK value [30]. After analysis, we finally decided to use average_received_bytes, average_received_ttl, average_received_ack_flags, average_received_push_flags, average_received_syn_flags, average_received_fin_flags, and average_received_window_size as the network-layer feature.…”