Using Lexical Features for Malicious URL Detection -- A Machine Learning Approach

Joshi, Apoorva; Lloyd, Levi; Westin, Paul; Seethapathy, Srini

doi:10.48550/arxiv.1910.06277

Cited by 5 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Examples of these features are: ratio of digits to letters in URL/hostname, URL contains an IP and number of subdomains. This category of features has been used for malicious URL detection [32] and for classifying DNS Type-squatting attacks [12].…”

Section: Lexical Featuresmentioning

confidence: 99%

Investigating the Influence of Feature Sources for Malicious Website Detection

Chaiban

Sovilj²,

Soliman³

et al. 2022

Applied Sciences

View full text Add to dashboard Cite

Malicious websites in general, and phishing websites in particular, attempt to mimic legitimate websites in order to trick users into trusting them. These websites, often a primary method for credential collection, pose a severe threat to large enterprises. Credential collection enables malicious actors to infiltrate enterprise systems without triggering the usual alarms. Therefore, there is a vital need to gain deep insights into the statistical features of these websites that enable Machine Learning (ML) models to classify them from their benign counterparts. Our objective in this paper is to provide this necessary investigation, more specifically, our contribution is to observe and evaluate combinations of feature sources that have not been studied in the existing literature—primarily involving embeddings extracted with Transformer-type neural networks. The second contribution is a new dataset for this problem, GAWAIN, constructed in a way that offers other researchers not only access to data, but our whole data acquisition and processing pipeline. The experiments on our new GAWAIN dataset show that the classification problem is much harder than reported in other studies—we are able to obtain around 84% in terms of test accuracy. For individual feature contributions, the most relevant ones are coming from URL embeddings, indicating that this additional step in the processing pipeline is needed in order to improve predictions. A surprising outcome of the investigation is lack of content-related features (HTML, JavaScript) from the top-10 list. When comparing the prediction outcomes between models trained on commonly used features in the literature versus embedding-related features, the gain with embeddings is slightly above 1% in terms of test accuracy. However, we argue that even this somewhat small increase can play a significant role in detecting malicious websites, and thus these types of feature categories are worth investigating further.

show abstract

Section: Lexical Featuresmentioning

confidence: 99%

Investigating the Influence of Feature Sources for Malicious Website Detection

Chaiban

Sovilj²,

Soliman³

et al. 2022

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…They stated that malicious URLs are delivered to users in different ways and that these URLs cause different harm to users. This study observed that machine learning methods were used to detect malicious URLs and an average of 92% accuracy value was obtained from 5 different data used for testing [15].…”

Section: 1related Workmentioning

confidence: 99%

Using and Comparing Machine Learning Techniques for Automatic Detection of Spam Website URLs

Yıldırım

2022

NATURENGS MTU Journal of Engineering and Natural Sciences Malatya Turgut Ozal University

View full text Add to dashboard Cite

With the developing technology, the issue of cyber security has become one of the most common and current issues in recent years. Spam URLs are one of the most common and dangerous issues for cybersecurity. Spam URLs are one of the most widely used attacks to defraud users. These attacks cause users to suffer monetary losses, steal private information, and install malicious software on their devices. It is very important to detect such threats promptly and to take precautions against these threats. Detection of malicious URLs is mostly done by using blacklists. However, these lists are insufficient to detect newly created URLs. In recent years, machine learning techniques have been developed to overcome this deficiency. In this study, URL classification was made using different machine learning techniques. In the study, 9 different classifiers were preferred for URL classification. The performances of the classifiers were compared in the URL classification process. In addition, similar studies in the literature have been comprehensively examined and these studies have been discussed. In addition, since the preparation of data sets in the natural language processing process has a great effect on the training of models, these steps are discussed in detail.

show abstract

“…The RData in a DNS response encompasses a list of resolved IP addresses, the time-to-live value of the query and the type of resource record. [8], [5], [4], [15], [26] [27], [6], [14], [13], [28] Side information features [29], [30], [16], [31], [32] our work Domain name string + [18], [33], [3], [34], [35] our work side information features [17], [10], [11], [12] TABLE I Overview of existing work on DGA detection ture would be "multi-valued". Alternatively, if the location could not be identified, then this feature takes the value "unknown".…”

Section: Side Informationmentioning

confidence: 99%

“…The value of the feature domain len for the domain name "google.com" is 10. [20], [14], [6], [13], [4], [26], [28], [11], [12] sld len Second level domain length [20], [14], [13] tld len Top level domain length [20], [14], [13] uni domain Domain Unique Characters length [20], [14], [13] uni sld SLD Unique Characters length [20], [14], [13] uni tld TLD Unique Characters length [20], [14], [13] flag dga Has malicious TLD [20], [14], [13], [26] tld hash TLD Hash [20], [14], [13], [6] flag dig…”

Section: Lexical Featuresmentioning

confidence: 99%

“…[20], [14], [13], [6] sym Symbol ratio [20], [14], [13], [6] hex Hex ratio [20], [14], [13], [6] dig Digit Ratio [20], [14], [13], [4], [28], [17], [11], [12] vow Vowel Ratio [20], [14], [13], [6], [4], [28] con Consonant Ratio [20], [14], [13] rep char ratio Ratio of Repeated Characters [20], [14], [4] cons con ratio Ratio of Consecutive Consonants [20], [14], [4], [28] cons dig ratio Ratio of Consecutive Digits [20], [14], [4] tokens sld Number of tokens in SLD [20], [14], [13], [26] digits sld Number of digits in SLD [20], [14], [13], [26] ent Entropy of characters in SLD [20], [14], …”

Section: Starts With Digitmentioning

confidence: 99%

See 1 more Smart Citation

Inline Detection of DGA Domains Using Side Information

Sivaguru

Peck

Olumofin³

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Malware applications typically use a command and control (C&C) server to manage bots to perform malicious activities. Domain Generation Algorithms (DGAs) are popular methods for generating pseudo-random domain names that can be used to establish a communication between an infected bot and the C&C server. In recent years, machine learning based systems have been widely used to detect DGAs. There are several well known state-of-the-art classifiers in the literature that can detect DGA domain names in real-time applications with high predictive performance. However, these DGA classifiers are highly vulnerable to adversarial attacks in which adversaries purposely craft domain names to evade DGA detection classifiers. In our work, we focus on hardening DGA classifiers against adversarial attacks. To this end, we train and evaluate stateof-the-art deep learning and random forest (RF) classifiers for DGA detection using side information that is harder for adversaries to manipulate than the domain name itself. Additionally, the side information features are selected such that they are easily obtainable in practice to perform inline DGA detection. The performance and robustness of these models is assessed by exposing them to one day of real-traffic data as well as domains generated by adversarial attack algorithms. We found that the DGA classifiers that rely on both the domain name and side information have high performance and are more robust against adversaries.

show abstract

Using Lexical Features for Malicious URL Detection -- A Machine Learning Approach

Cited by 5 publications

References 0 publications

Investigating the Influence of Feature Sources for Malicious Website Detection

Investigating the Influence of Feature Sources for Malicious Website Detection

Using and Comparing Machine Learning Techniques for Automatic Detection of Spam Website URLs

Inline Detection of DGA Domains Using Side Information

Contact Info

Product

Resources

About