A survey of bitmap index compression algorithms for Big Data

Chen, Zhen; Wen, Yang; Cao, Junwei; Zheng, Wenxun; Chang, Jian; Wu, Yinjun; Ma, Ge; Hakmaoui, Mourad; Peng, Guodong

doi:10.1109/tst.2015.7040519

Cited by 33 publications

(4 citation statements)

References 48 publications

(21 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the literature, researchers use different data structures to organize flow (or query) indexes [7]. For example, TelegraphCQ [5] stores indexes and results of queries via a modified version of PostgreSQL.…”

Section: Flow-level Traffic Collectorsmentioning

confidence: 99%

CompactFlow: A Hybrid Binary Format for Network Flow Data

Piskozub¹,

Spolaor²,

Martinovic³

2020

Information Security Theory and Practice

View full text Add to dashboard Cite

Network traffic monitoring has become fundamental to obtaining insights about a network and its activities. This knowledge allows network administrators to detect anomalies, identify faulty hardware, and make informed decisions. The increase of the number of connected devices and the consequent volume of traffic poses a serious challenge to carrying out the task of network monitoring. Such a task requires techniques that process traffic in an efficient and timely manner. Moreover, it is crucial to be able to store network traffic for forensic purposes for as long a period of time as possible. In this paper, we propose CompactFlow, a hybrid binary format for efficient storage and processing of network flow data. Our solution offers a trade-off between the space required and query performance via an optimized binary representation of flow records and optional indexing. We experimentally assess the efficiency of CompactFlow by comparing it to a wide range of binary flow storage formats. We show that Compact-Flow format improves the state of the art by reducing the size required to store network flows by more than 24%.

show abstract

Section: Flow-level Traffic Collectorsmentioning

confidence: 99%

CompactFlow: A Hybrid Binary Format for Network Flow Data

Piskozub¹,

Spolaor²,

Martinovic³

2020

Information Security Theory and Practice

View full text Add to dashboard Cite

show abstract

“…Several approaches have been demonstrated to speed up searching operations, such as parallel processing, materialized views, and indexing [4][5][6][7][8][9]. Among these three, indexing is an efficient approach to retrieve data without requiring additional hardware and to enable answering ad hoc and complex queries in reasonable times [7,[10][11][12]. One indexing technique for enabling fast query processing over large read-only repositories in data warehouses is bitmap-based indexes [5,10,11,13,14].…”

Section: Introductionmentioning

confidence: 99%

“…Among these three, indexing is an efficient approach to retrieve data without requiring additional hardware and to enable answering ad hoc and complex queries in reasonable times [7,[10][11][12]. One indexing technique for enabling fast query processing over large read-only repositories in data warehouses is bitmap-based indexes [5,10,11,13,14]. The common bitmap-based index is known as a basic bitmap index [15].…”

Section: Introductionmentioning

confidence: 99%

HyBiX: A novel encoding bitmap index for space- and time-efficient query processing

Keawpibal¹,

Preechaveerakul²,

Vanichayobon³

2019

Turk J Elec Eng & Comp Sci

View full text Add to dashboard Cite

A bitmap-based index is an effective and efficient indexing method for answering selective queries in a read- only environment. It offers improved query execution time by applying low-cost Boolean operators on the index directly, before accessing raw data. A drawback of the bitmap index is that index size increases with the cardinality of indexed attributes, which additionally has an impact on a query execution time. This impact is related to an increase of query execution time due to the scanning of bitmap vectors to answer the queries. In this paper, we propose a new encoding bitmap index, called the HyBiX bitmap index. The HyBiX bitmap index was experimentally compared to existing encoding bitmap indexes in terms of space requirement, query execution time, and space and time trade-off for equality and range queries. As experimental results, the HyBiX bitmap index can reduce space requirements with high cardinality attributes with satisfactory execution times for both equality and range queries. The performance of the HyBiX bitmap index provides the second-best results for equality queries and the first-best for range queries in terms of space and time trade-off.

show abstract

“…Some of them are already applied in network traffic archival and retrieval. In Reference [22], Chen et al presented a survey of different traffic archival and retrieval systems and various bitmap index compression algorithms.…”

mentioning

confidence: 99%

An Efficient Indexing Scheme for Network Traffic Collection and Retrieval System

2021

View full text Add to dashboard Cite

Historical network traffic retrieval, both at the packet and flow level, has been applied in many fields of network security, such as network traffic analysis and network forensics. To retrieve specific packets from a vast number of packet traces, it is an effective solution to build indexes for the query attributes. However, it brings challenges of storage consumption and construction time overhead for packet indexing. To address these challenges, we propose an efficient indexing scheme called IndexWM based on the wavelet matrix data structure for packet indexing. Moreover, we design a packet storage format based on the PcapNG format for our network traffic collection and retrieval system, which can speed up the extraction of index data from packet traces. Offline experiments on randomly generated network traffic and actual network traffic are performed to evaluate the performance of the proposed indexing scheme. We choose an open-source and widely used bitmap indexing scheme, FastBit, for comparison. Apart from the native bitmap compression method Word-Aligned Hybrid (WAH), we implement an efficient bitmap compression method Scope-Extended COMPAX (SECOMPAX) in FastBit for performance evaluation. The comparison results show that our scheme outperforms the selected bitmap indexing schemes in terms of time consumption, storage consumption and retrieval efficiency.

show abstract

A survey of bitmap index compression algorithms for Big Data

Cited by 33 publications

References 48 publications

CompactFlow: A Hybrid Binary Format for Network Flow Data

CompactFlow: A Hybrid Binary Format for Network Flow Data

HyBiX: A novel encoding bitmap index for space- and time-efficient query processing

An Efficient Indexing Scheme for Network Traffic Collection and Retrieval System

Contact Info

Product

Resources

About