A Survey on Networked Data Streaming With Apache Kafka

Raptis, Theofanis P.; Passarella, Andrea

doi:10.1109/access.2023.3303810

Cited by 15 publications

(4 citation statements)

References 72 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It can ingest streaming data from the distributed source to the stream processing engine for analytics. We choose Apache Kafka for its high-throughout, fault-tolerant, distributed, and scalable nature, making it well-suited for handling the ever-growing streaming traffic from IoT devices [50]. In our proposed system, Apache Kafka ingests the flow-based statistical features of each flow from NFStream to the online network stream analyzer component in regular time intervals.…”

Section: ) Apache Kafkamentioning

confidence: 99%

UASDAC: An Unsupervised Adaptive Scalable DDoS Attack Classification in Large-Scale IoT Network Under Concept Drift

Selvam,

Maheswari Balasubramanian

2024

IEEE Access

View full text Add to dashboard Cite

Day by day, the number of devices in IoT networks is increasing, and concurrently, the size of botnets in IoT networks is also expanding. Currently, attackers prefer IoT-based botnets to launch DDoS attacks, as IoT devices offer a vast attack surface. Many researchers have proposed machine and deep learning-based classifiers to classify DDoS and benign network traffic in online streams from IoT devices. However, the performance of the traditional machine and deep learning algorithms deteriorates when sudden concept or data drift occurs in the online streams and the volume and velocity of IoT network traffic increases. To address these challenges, we propose UASDAC, an adaptive and scalable data pipeline designed specifically to handle concept drift and detect DDoS traffic in real-time in massive online streams originating from IoT devices. UASDAC incorporates three key components: an online network stream collector for data collection, an online network stream analyzer with an unsupervised drift detector for detecting drift and DDoS traffic, and an online network stream repository for storing streams for future analytics. UASDAC leverages big data technologies to implement all the three components to achieve scalability. Additionally, UASDAC introduces an effective and efficient retraining technique to adapt to novel patterns in online streams in the presence of concept drift. We evaluated the performance of UASDAC in different concept drift scenarios using the benchmark dataset NSL-KDD and the latest IoT dataset IoT23. Our results demonstrate that UASDAC effectively identifies DDoS traffic in the presence of concept drift, achieving an accuracy range of 99.7% to 99.9%.

show abstract

Section: ) Apache Kafkamentioning

confidence: 99%

UASDAC: An Unsupervised Adaptive Scalable DDoS Attack Classification in Large-Scale IoT Network Under Concept Drift

Selvam,

Maheswari Balasubramanian

2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…For the implementation of the Fabric Integration features, including facilitating interoperability and communication between management services (within and between domains), service mesh solutions such as Istio [118] or Linkerd [119] were analyzed, as well as an event streaming platform, like Apache Kafka [120]. On the one hand, the service mesh will allow the management of traffic between services, while enhancing security and observability.…”

Section: Autonomic and Cognitive Security Managementmentioning

confidence: 99%

Security in Cloud-Native Services: A Survey

Theodoropoulos,

Rosa,

Benzaid

et al. 2023

JCP

View full text Add to dashboard Cite

Cloud-native services face unique cybersecurity challenges due to their distributed infrastructure. They are susceptible to various threats like malware, DDoS attacks, and Man-in-the-Middle (MITM) attacks. Additionally, these services often process sensitive data that must be protected from unauthorized access. On top of that, the dynamic and scalable nature of cloud-native services makes it difficult to maintain consistent security, as deploying new instances and infrastructure introduces new vulnerabilities. To address these challenges, efficient security solutions are needed to mitigate potential threats while aligning with the characteristics of cloud-native services. Despite the abundance of works focusing on security aspects in the cloud, there has been a notable lack of research that is focused on the security of cloud-native services. To address this gap, this work is the first survey that is dedicated to exploring security in cloud-native services. This work aims to provide a comprehensive investigation of the aspects, features, and solutions that are associated with security in cloud-native services. It serves as a uniquely structured mapping study that maps the key aspects to the corresponding features, and these features to numerous contemporary solutions. Furthermore, it includes the identification of various candidate open-source technologies that are capable of supporting the realization of each explored solution. Finally, it showcases how these solutions can work together in order to establish each corresponding feature. The insights and findings of this work can be used by cybersecurity professionals, such as developers and researchers, to enhance the security of cloud-native services.

show abstract

“…In recent works [3,4], Kafka has been actively used to monitor data and store the history of additions.…”

Section: Introductionmentioning

confidence: 99%

Designing a Data Warehouse for Collected Data About User Activity in Social Networks Using Elasticsearch

Mysiuk

2023

PoS

View full text Add to dashboard Cite

In this paper, a data storage data warehouse is designed to store collected data from social networks. Creating indexes with data and selecting a configuration with the appropriate number of shards and replicas is described -the primary states of the cluster and possibilities of its scaling. The features of working with the non-relational Elasticsearch database are described when working with data on user activity in social network posts. Among social networks, Facebook and Instagram were chosen for analysis. The paper describes the advantages and disadvantages of using such a data store compared to Apache Kafka. Analysed existing data insertion Application Program Interfaces (APIs) and data visualisation tools integrated with Elasticsearch. The study describes the use of the Bulk API to insert many records at once into a database. The designed data warehouse uses Kibana, a data visualisation and analytics tool integrated with the selected database. Also, it is shown the ability to insert and view logs using Elasticsearch, Logstash, and Kibana (ELK stack). Tested data ingest by logging into the database using Beats. The obtained results can help implement a system for analysing user activities from social network data based on Elasticsearch as a central component.

show abstract

A Survey on Networked Data Streaming With Apache Kafka

Cited by 15 publications

References 72 publications

UASDAC: An Unsupervised Adaptive Scalable DDoS Attack Classification in Large-Scale IoT Network Under Concept Drift

UASDAC: An Unsupervised Adaptive Scalable DDoS Attack Classification in Large-Scale IoT Network Under Concept Drift

Security in Cloud-Native Services: A Survey

Designing a Data Warehouse for Collected Data About User Activity in Social Networks Using Elasticsearch

Contact Info

Product

Resources

About