Resource-adaptive real-time new event detection

Abstract

In a document streaming environment, online detection of the first documents that mention previously unseen events is an open challenge. For this online new event detection (ONED) task, existing studies usually assume that enough resources are always available and focus entirely on detection accuracy without considering efficiency. Moreover, none of the existing work addresses the issue of providing an effective and friendly user interface. As a result, there is a significant gap between the existing systems and a system that can be used in practice. In this paper, we propose an ONED framework with the following prominent features. First, a combination of indexing and compression methods is used to improve the document processing rate by orders of magnitude without sacrificing much detection accuracy. Second, when resources are tight, a resource-adaptive computation method is used to maximize the benefit that can be gained from the limited resources. Third, when the new event arrival rate is beyond the processing capability of the consumer of the ONED system, new events are further filtered and prioritized before they are presented to the consumer. Fourth, implicit citation relationships are created among all the documents and used to compute the importance of document sources. This importance information can guide the selection of document sources. We implemented a prototype of our framework on top of IBM's Stream Processing Core middleware. We also evaluated the effectiveness of our techniques on the standard TDT5 benchmark. To the best of our knowledge, this is the first implementation of a real application in a large-scale stream processing system.

An Improved System for Sentence-level Novelty Detection in Textual Streams

“…Some work has been carried out by Song et al [5] to improve the efficiency of novelty detection systems by introducing a news indexing-tree. [6] presents a framework for online new event detection used in a real application. The approach used is called 1-NN approach and the framework focuses on improving system efficiency by reducing the number of saved documents using indices, parallel processing, and etc.…”

Maintaining dynamic channel profiles on the web

“…For this purpose, we suggest a learning scheme based on novelty detection for Web channel monitoring in a pullonly setting. Both traditional novelty detection solutions (e.g., [1,16,29]), and state-of-the-art news summarization (e.g., [21]), news filtering (e.g., [2,10]), and topic tracking (e.g., [20]) solutions assume that the complete stream of update events is available via push into the system. Furthermore, the main focus of these solutions is on efficient data filtering and novelty detection methods, where load shedding techniques can be applied in order to control the amount of pushed data to be consumed by the system (e.g., [28]).…”

Bilkent news portal

“…It contains more than 200,000 news and 80 events annotated by 39 native speakers. In the system implementation, for the event detection sub-component we employ the time window concept [3] and some novel approaches such as combined similarity measures.…”