Performance Prediction for the Apache Kafka Messaging System

Wu, Han; Shang, Zhihao; Wolter, Katinka

doi:10.1109/hpcc/smartcity/dss.2019.00036

Cited by 38 publications

(17 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…4, we noted different periods where the frequency of messages is higher than those of other periods. In particular, we note the 00 minutes, minutes 30-33 and minutes [43][44][45][46] appear to represent busier periods, which could contribute to correlation in our data. We investigated the impact of the removal of these scheduled messages from our data.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Heads Or Tails: A Framework To Model Supply Chain Heterogeneous Messages

Leech

Malone

Dunne

2021

2021 30th Conference of Open Innovations Association FRUCT

View full text Add to dashboard Cite

The electronic exchange of business to business information (e.g. purchase orders, inventory data and shipment notices between departments or organizations) can eliminate the need for human intervention and paper copy trails. Incorporating Electronic Data Interchange (EDI) standards into an organization can drastically improve the efficiency of processing times. Modelling the behaviour of EDI messages within a Supply Chain network's queuing system has many purposes, from understanding the efficiency of queue behaviour to process reengineering. In this paper we demonstrate that these messages are heterogeneous, suffer from correlation, are not stationary and are challenging to model. We investigate whether a parametric or non-parametric approach is appropriate to model message service and inter-arrival times. Our results show that parametric distribution models are suitable for modelling the distribution's tail, whilst non-parametric Kernel Density Estimation models are better suited for modelling the head.

show abstract

Section: Resultsmentioning

confidence: 99%

“…Wu, Shang & Wolter analyzed the performance of Kafka using specific tuning parameters [46]. Their analysis concluded that the performance varied significantly depending on message type and system infrastructure.…”

Section: Related Workmentioning

confidence: 99%

Heads Or Tails: A Framework To Model Supply Chain Heterogeneous Messages

Leech

Malone

Dunne

2021

2021 30th Conference of Open Innovations Association FRUCT

View full text Add to dashboard Cite

show abstract

“…This component is highly critical since each incoming event passes through it. The decision for Apache Kafka was made due its high-throughput, low-latency, resiliency and scalability [28]. As shown in Figure 3, two internal components are listening to the message queue for incoming events: the scope streamer and the storage sink.…”

Section: Architecturementioning

confidence: 99%

Scalable Fleet Monitoring and Visualization for Smart Machine Maintenance and Industrial IoT Applications

Moens

Bracke

Soete

et al. 2020

Sensors

View full text Add to dashboard Cite

The wide adoption of smart machine maintenance in manufacturing is blocked by open challenges in the Industrial Internet of Things (IIoT) with regard to robustness, scalability and security. Solving these challenges is of uttermost importance to mission-critical industrial operations. Furthermore, effective application of predictive maintenance requires well-trained machine learning algorithms which on their turn require high volumes of reliable data. This paper addresses both challenges and presents the Smart Maintenance Living Lab, an open test and research platform that consists of a fleet of drivetrain systems for accelerated lifetime tests of rolling-element bearings, a scalable IoT middleware cloud platform for reliable data ingestion and persistence, and a dynamic dashboard application for fleet monitoring and visualization. Each individual component within the presented system is discussed and validated, demonstrating the feasibility of IIoT applications for smart machine maintenance. The resulting platform provides benchmark data for the improvement of machine learning algorithms, gives insights into the design, implementation and validation of a complete architecture for IIoT applications with specific requirements concerning robustness, scalability and security and therefore reduces the reticence in the industry to widely adopt these technologies.

show abstract

“…Clearly, it is not recommended for production. On the other hand, Kafka uses a fault-tolerant distributed streaming platform called Apache Kafka [46]. It also enables distributed ordering service, so that we can have multiple Orderer nodes to avoid a single point of failure [45], [47].…”

Section: ) Hyperledger Fabric Networkmentioning

confidence: 99%

Demand Response Service Certification and Customer Baseline Evaluation Using Blockchain Technology

et al. 2020

View full text Add to dashboard Cite

The use of Distributed Ledger Technologies such as Blockchain for certifying Demand Response services allows for the creation of a distributed system in which customers can communicate with the system operator to provide their flexibility, in a secure, transparent and traceable way. Blockchain technology also supports incentive mechanisms for users taking part in the service through the generation of utility tokens to recognize the user's contribution. This paper presents the experimental test of a novel methodology for Demand Response programs implementation by using the Blockchain technology. The latter is employed for defining a distributed Demand Response service and a new system for its tracing and certification. For this work, a Smart Contract has been conceived and written to execute Demand Response events, calculate users' baseline, compute the support provided by each user towards the fulfilment of the requested load curve modification and remunerate each user with utility tokens proportionally to their contribution. To test the methodology, a Hyperledger Fabric network and a Smart Contract were deployed on four nodes of the Microgrid Laboratory of the Department of Energy Technology at Aalborg University (DK). Subsequently, a realistic scenario comprising two consumer nodes was developed using power electronic converters for generating the household profiles and Smart Meters for the measurement of the consumption profiles. Theoretical and experimental results show the feasibility of Distributed Ledger Technologies in smart grids management with a minimum investment in new hardware while enabling the active participation of customers in Demand Response more transparently and fairly.

show abstract

Performance Prediction for the Apache Kafka Messaging System

Cited by 38 publications

References 12 publications

Heads Or Tails: A Framework To Model Supply Chain Heterogeneous Messages

Heads Or Tails: A Framework To Model Supply Chain Heterogeneous Messages

Scalable Fleet Monitoring and Visualization for Smart Machine Maintenance and Industrial IoT Applications

Demand Response Service Certification and Customer Baseline Evaluation Using Blockchain Technology

Contact Info

Product

Resources

About