Importance of web security cannot be overemphasized in the era of web-based economy. Although anomaly detection has long been considered a promising alternative to signature-based misuse detection technique, most studies to date used either small scale or artificially generated attack data. In this paper, based on security analysis applied on anonymous www.microsoft.com log of about 250GB, we propose Anomaly Feature Matrix (AFM) as an effective framework to characterize anomalies. Feature selection of AFM is based on the characteristics of well-known (e.g., DDoS) attacks as well as patterns of anomalous logs found in the Microsoft data. Independent security analysis performed on the same data by Microsoft security engineers concluded that 1) We did not miss any major attacks; and 2) AFM is a general enough framework to characterize likely web attacks. In order to assist AFM-based anomaly analysis in large organizations, we implemented an interactive and visual analysis tool named ADAM (Anomaly Detection Assistant based on feature Matrix). Integrated with mapping software such as Virtual Earth, ADAM enables efficient and focused security analysis on web logs.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.