Efficient Join Processing Over Incomplete Data Streams

Abstract

For decades, the join operator over fast data streams has always drawn much attention from the database community, due to its wide spectrum of real-world applications, such as online clustering, intrusion detection, sensor data monitoring, and so on. Existing works usually assume that the underlying streams to be joined are complete (without any missing values). However, this assumption may not always hold, since objects from streams may contain some missing attributes, due to various reasons such as packet losses, network congestion/failure, and so on. In this paper, we formalize an important problem, namely join over incomplete data streams (Join-iDS), which retrieves joining object pairs from incomplete data streams with high confidences. We tackle the Join-iDS problem in the style of "data imputation and query processing at the same time". To enable this style, we design an effective and efficient cost-modelbased imputation method via deferential dependency (DD), devise effective pruning strategies to reduce the Join-iDS search space, and propose efficient algorithms via our proposed cost-model-based data synopsis/indexes. Extensive experiments have been conducted to verify the efficiency and effectiveness of our proposed Join-iDS approach on both real and synthetic data sets. Figure 1 illustrates two critical routers, O and U , in an IP network, from which we collect statistical (log) attributes in a streaming manner, for example, No. of connections, the connection duration, and the transferred data size. In practice, due to packet losses, network congestion/delays, or hardware failure, we may not always obtain all attributes from each router. As an example in Table 1, the transferred data size of router o t is missing (denoted as "-") at timestamp t. As a result, stream data collected from each router may sometimes contain incomplete attributes. One critical, yet challenging, problem in the network is to monitor network traffic, and detect potential network intrusion. If one router (e.g., O) is under the attack of network intrusion, we should quickly identify potential attacks in other routers, like U , at close timestamps, to which we may take actions for protecting the network security. In

Data Stream Summary in Big Data Context: Challenges and Opportunities

“…As another sampling method, we have the join sampling [31], [32] which attempts to connect distributed data streams, for example, from meteorological measurement sensors from several stations. This method has the advantage of drawing each event in the sample in a single pass.…”

Online Topic-Aware Entity Resolution Over Incomplete Data Streams

“…In this paper, based on complete data repositories, we adopt CDDs as our imputation technique for imputing missing attributes. Moreover, there are some works on queries over incomplete data streams, such as join, skyline, and top-𝑘 operators [30][31][32].…”