Malicious URL filtering &amp;#x2014; A big data application

Lin, Min-Sheng; Chiu, Chien-Yi; Lee, Yuh-Jye; Pao, Hsing-Kuo

doi:10.1109/bigdata.2013.6691627

Cited by 43 publications

(57 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The proposed schemes can be classified into either static or dynamic detection systems. Some lightweight static detection systems focus on the lexical features of a URL [19], as well as DNS and WHOIS information [11]. And other static detection systems extract additional features from the HTML content and JavaScript codes [7].…”

Section: Related Workmentioning

confidence: 99%

GuidedTracker: Track the victims with access logs to finding malicious web pages

Sha

Liu

Zhou

et al. 2014

2014 IEEE Global Communications Conference

View full text Add to dashboard Cite

Malicious web pages have become a malignant tumour for the Internet, which spread malicious code, steal people's private information, and deliver spamming advertisements. And how to distinguish them from the huge number of normal web pages effectively remains a huge challenge in the era of big data. To detect malicious pages, one needs to first collect candidate web pages that are live on the web; then filter massive legitimate pages using fast filters and finally examine the remaining pages using precisely but slow analyzer. However, there are new challenges recently for these conventional techniques, including large scale, imbalance data and the usage of cloaking techniques. To cope with these challenges, the malicious URL detection system should perform more efficiently.In this paper, we propose a system, named GuidedTracker, to search for suspicious malicious pages. GuidedTracker starts from the seed set which includes known malicious pages. Then, it automatically figures out those victims based on the seed set and the visit relation database. Finally, the access records of these victims are used to identify other malicious pages. In this way, GuidedTracker increase the percentages of malicious URLs in the input URL stream submitted to the precisely analyzer. To our best knowledge, GuidedTracker is the first to introduce visit relations to tackle the malicious URL detection problem. The introduction of visit relations limits the scope of URL inspection and enables this approach to have the ability of self-learning. Experimental results show that the overall "toxicity" can be improved by 6.97% -50.38% compared with full inspection of access logs.

show abstract

Section: Related Workmentioning

confidence: 99%

GuidedTracker: Track the victims with access logs to finding malicious web pages

Sha

Liu

Zhou

et al. 2014

2014 IEEE Global Communications Conference

View full text Add to dashboard Cite

show abstract

“…The research on new methods for sketching, anomaly detection, noise removal, feature extraction, outliers detection, and pre-filtering of streaming data is required to reduce big data effectively. In addition, the deployment of adaptive learning models in conjunction with said methods can aid in dynamic preprocessing of big streaming data [21].…”

Section: Open Research Issuesmentioning

confidence: 99%

“…The authors in [21] proposed two feature reduction techniques that extract the lexical features and the descriptive features and then combine their results. The lexical features extract the structure of the URLs.…”

Section: Data Preprocessingmentioning

confidence: 99%

“…The aforementioned methods first extract the semantics and linked structures from the unstructured datasets and then apply graph theory for network optimization. Conversely, some methods to reduce big data during the data collection process are also proposed in the recent literature [19][20][21]. In this study, we presented a detailed discussion of these data reduction methods.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Big Data Reduction Methods: A Survey

et al. 2016

View full text Add to dashboard Cite

Research on big data analytics is entering in the new phase called fast data where multiple gigabytes of data arrive in the big data systems every second. Modern big data systems collect inherently complex data streams due to the volume, velocity, value, variety, variability, and veracity in the acquired data and consequently give rise to the 6Vs of big data. The reduced and relevant data streams are perceived to be more useful than collecting raw, redundant, inconsistent, and noisy data. Another perspective for big data reduction is that the million variables big datasets cause the curse of dimensionality which requires unbounded computational resources to uncover actionable knowledge patterns. This article presents a review of methods that are used for big data reduction. It also presents a detailed taxonomic discussion of big data reduction methods including the network theory, big data compression, dimension reduction, redundancy elimination, data mining, and machine learning methods. In addition, the open research issues pertinent to the big data reduction are also highlighted.

show abstract

“…Chu et al [12] used only lexical and domain features of URLs to detect phishing sites based on machine learning methods. Lin et al [13] generated online learning algorithms using lexical features and descriptive features working on large scale URL dataset to reduce the volume of URL queries on which further analysis needs to be performed. Huang et al [14] propose to dynamically extract lexical patterns from URLs instead of using any pre-defined features or fixed delimiters for feature selection.…”

Section: Related Workmentioning

confidence: 99%

Detection of Delivery Servers in Online Video Service using Long Short Term Memory Network

Zheng¹,

Li²,

Lin³

2018

dtcse

View full text Add to dashboard Cite

Nowadays large scale video delivery networks have been widely deployed. Information about video delivery servers is crucial for a series of network management tasks. However, it is a challenge to identify the delivery servers in the ever-changing content delivery networks. Moreover, it is difficult to find general and flexible features to identify video delivery servers from various video service providers. In this paper, we mainly come up with a novel method based on LSTM (Long Short Term Memory Network) to detect video delivery servers. Experimental results prove that our approach works pretty well by comparing it with the other two conventional machine learning methods. What's more, our approach can work more flexibly without cumbersome feature extraction relative to conventional machine learning methods.

show abstract

Malicious URL filtering — A big data application

Cited by 43 publications

References 9 publications

GuidedTracker: Track the victims with access logs to finding malicious web pages

GuidedTracker: Track the victims with access logs to finding malicious web pages

Big Data Reduction Methods: A Survey

Detection of Delivery Servers in Online Video Service using Long Short Term Memory Network

Contact Info

Product

Resources

About