Real-time Event Detection on Social Data Streams

Abstract

Social networks are quickly becoming the primary medium for discussing what is happening around real-world events. The information that is generated on social platforms like Twitter can produce rich data streams for immediate insights into ongoing matters and the conversations around them. To tackle the problem of event detection, we model events as a list of clusters of trending entities over time. We describe a real-time system for discovering events that is modular in design and novel in scale and speed: it applies clustering on a large stream with millions of entities per minute and produces a dynamically updated set of events. In order to assess clustering methodologies, we build an evaluation dataset derived from a snapshot of the full Twitter Firehose and propose novel metrics for measuring clustering quality. Through experiments and system profiling, we highlight key results from the offline and online pipelines. Finally, we visualize a high profile event on Twitter to show the importance of modeling the evolution of events, especially those detected from social data streams.

Follow the Blue Bird: A Study on Threat Data Published on Twitter

“…It is hard to describe all relevant elements of a large company thoroughly, and maybe not all software in use is indexed or known. By extending the collection elements with (for example) topic detection techniques (e.g., [33,38,46]), one is more likely to cover all software in use. As Twitter can provide all these types of news and the research community has studied thoroughly topic detection on this platform, having trend detection might be mandatory for effective OSINT collection.…”

Cybersecurity Event Detection with New and Re-emerging Words

“…Weng and Lee [47] proposed a method by applying wavelet transformation on word-specific signals, such as df-idf scores of words in time domain, to filter out trivial words and then clustering the remaining words to discover events. In a similar manner to Fung et al [25] and Mathioudakis and Koudas [32], Fedoryszak et al [24] proposed a real-time event detection system that discovers events whose occurrences are different from normal levels of conversations. It is natural to consider applying general-purpose event detection methods in the literature to cybersecurity domain.…”

“…At the beginning of new malware appearance, there are usually few mentions about it. However, they neither provide how to handle new words [25] nor detect cyber threats whose volume is small [24,32,47]. In addition, the existing methods [24,25,32] sometimes fail to detect resurgent threats like malware variants in their early stages.…”

“…However, they neither provide how to handle new words [25] nor detect cyber threats whose volume is small [24,32,47]. In addition, the existing methods [24,25,32] sometimes fail to detect resurgent threats like malware variants in their early stages. Since they identified bursty words for event detection by measuring the deviance of the observed occurrences from the expected occurrences computed based on their historical appearance proportions in Twitter stream, they fail to detect events that had been mentioned in large scale earlier, but emerge again with small volume.…”

“…Since they identified bursty words for event detection by measuring the deviance of the observed occurrences from the expected occurrences computed based on their historical appearance proportions in Twitter stream, they fail to detect events that had been mentioned in large scale earlier, but emerge again with small volume. For example, when a new Mirai variant, called Wicked, appeared on May 17th 2018, the bursty event detection methods [24,25,32] fail to detect it with the word "mirai". This is because "mirai" had appeared over months with moderate to large number of mentions due to its several variants, so its expected occurrence was higher than its observed occurrence (8 tweets) at the time of emergence of the Wicked botnet.…”