inTIME: A Machine Learning-Based Framework for Gathering and Leveraging Web Data to Cyber-Threat Intelligence

Abstract

In today’s world, technology has become deep-rooted and more accessible than ever over a plethora of different devices and platforms, ranging from company servers and commodity PCs to mobile phones and wearables, interconnecting a wide range of stakeholders such as households, organizations and critical infrastructures. The sheer volume and variety of the different operating systems, the device particularities, the various usage domains and the accessibility-ready nature of the platforms creates a vast and complex threat landscape that is difficult to contain. Staying on top of these evolving cyber-threats has become an increasingly difficult task that presently relies heavily on collecting and utilising cyber-threat intelligence before an attack (or at least shortly after, to minimize the damage) and entails the collection, analysis, leveraging and sharing of huge volumes of data. In this work, we put forward inTIME, a machine learning-based integrated framework that provides an holistic view in the cyber-threat intelligence process and allows security analysts to easily identify, collect, analyse, extract, integrate, and share cyber-threat intelligence from a wide variety of online sources including clear/deep/dark web sites, forums and marketplaces, popular social networks, trusted structured sources (e.g., known security databases), or other datastore types (e.g., pastebins). inTIME is a zero-administration, open-source, integrated framework that enables security analysts and security stakeholders to (i) easily deploy a wide variety of data acquisition services (such as focused web crawlers, site scrapers, domain downloaders, social media monitors), (ii) automatically rank the collected content according to its potential to contain useful intelligence, (iii) identify and extract cyber-threat intelligence and security artifacts via automated natural language understanding processes, (iv) leverage the identified intelligence to actionable items by semi-automatic entity disambiguation, linkage and correlation, and (v) manage, share or collaborate on the stored intelligence via open standards and intuitive tools. To the best of our knowledge, this is the first solution in the literature to provide an end-to-end cyber-threat intelligence management platform that is able to support the complete threat lifecycle via an integrated, simple-to-use, yet extensible framework.

A Shared Cyber Threat Intelligence Solution for SMEs

“…In our literature review, we found that researchers are starting to apply supervised machine learning [61], natural language processing (NLP) [47,55], and deep learning [52] techniques to process shared CTI. However, we also found that applying an expert evaluation to the raw data, or using production rules, was far more popular.…”

Threats from the Dark: A Review over Dark Web Investigation Research for Cyber Threat Intelligence

“…e INTIME tool in Koloveas et al's work [32] provides a framework to identify and analyze cyber threats in specific cybersecurity topics (Internet of ings (IoT)) to share knowledge among cybersecurity organizations.…”

An enhanced network intrusion detection system for malicious crawler detection and security event correlations in ubiquitous banking infrastructure

“…Following it, the crawlers try to use the Web mining methods. Finally, crawlers perform distributed denial of service and remove the sites for quite a while (Vishwakarma and Jain, 2020; Rahman et al , 2020; Koloveas et al , 2021; Xu et al , 2018). It is disagreeable in any circumstance, especially in banking, as the data is sensitive.…”