Large Scale Image Processing in Real-Time Environments with Kafka

Kim, Yoon-Ki; Jeong, Chang-Sung

doi:10.5121/csit.2017.70120

Cited by 6 publications

(2 citation statements)

References 2 publications

(2 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Kafka is widely used for building realtime data pipelines and streaming applications, such as log aggregation, event-driven architectures, and stream processing [22]. Some key features of Kafka include a publish-subscribe model, fault tolerance, and horizontal scalability, which make it suitable for large-scale, real-time data processing tasks [20].…”

Section: Apache Kafkamentioning

confidence: 99%

Exploring Real-Time Sentiment Analysis Prototype for Retail Industry

Yuen,

Juddoo

2023

iCARTi

View full text Add to dashboard Cite

The rise of social media platforms has revolutionized the way consumers interact with retailers and express their opinions on products and services. Online retailers particularly need to keep a close eye on customer sentiment in real-time to make informed decisions about their offerings and improve customer satisfaction. However, efficiently analysing large volumes of unstructured text data from social media in real-time poses a significant challenge. This research aimed to develop a scalable, real-time sentiment analysis system tailored for online retailers using Reddit as the data source. The system comprises three main components: a data extraction and streaming pipeline, a sentiment analysis model, and a web application with real-time analytics. To address the data extraction challenge, a job queue-based system was implemented using Node.js, ‘BullMQ’, and Redis to create and manage campaigns for data streaming from Reddit. The data was streamed using Kafka, a distributed streaming platform, to enable efficient real-time processing. The sentiment analysis model was developed using a Naive Bayes classifier after experimenting with other machine learning and deep learning techniques. In the conducted study, the sentiment analysis model's performance was evaluated using standard metrics tailored to the context of online retail sentiment analysis. An accuracy of 0.6737 was achieved, reflecting the model's ability to correctly classify approximately 67.37 per cent of the sentiments in the test data. Concurrently, an F1 score of 0.7894 was recorded and the Area Under the Curve (AUC) value on the test data was measured at 0.5468, a metric that, while acceptable, suggests room for further refinement in the model's discriminatory ability between classes. The integration of the Data Version Control (DVC) system provided a mechanism for fine-tuning the model according to specific data requirements of various tenants. These results, taken together, not only validate the feasibility of employing a Naive Bayes classifier for real-time sentiment analysis in the retail context, but also provide a baseline for future research aimed at enhancing both the accuracy and efficiency of sentiment classification. The project’s evaluation focused on the performance of the sentiment analysis model, the efficiency of the Kafka streaming and real-time Spark pre-processing pipeline, and the backend infrastructure, including the job queuing system and WebSocket implementation. Various evaluation techniques, such as graphs and literature comparisons, were used to assess the system’s performance. In conclusion, this project successfully demonstrated the feasibility of a scalable, real-time sentiment analysis system for online retailers using Reddit data. The system has the potential to help retailers better understand customer opinions and make data-driven decisions for their businesses. Future work could include exploring alternative data sources, experimenting with more advanced sentiment analysis techniques, and enhancing the web application’s user interface and analytics capabilities.

show abstract

Section: Apache Kafkamentioning

confidence: 99%

Exploring Real-Time Sentiment Analysis Prototype for Retail Industry

Yuen,

Juddoo

2023

iCARTi

View full text Add to dashboard Cite

show abstract

“…RIDE provides a user friendly programming environment by supporting coarse-grained parallelism automatically by the system while the users only need to consider fine-grained parallelism by careful parallel programming on multicore or GPU. For real-time massive stream image processing, we design a distributed buffer system based on Kafka [12] which enables each of distributed nodes to access and process the buffered image in parallel [13], improving its overall performance sharply. Besides, it supports dynamic allocation of partitions to worker nodes which maximizes the throughput by preventing worker nodes from being idle.…”

Section: Introductionmentioning

confidence: 99%

RIDE: real-time massive image processing platform on distributed environment

Kim

Jeong

2018

J Image Video Proc.

Self Cite

View full text Add to dashboard Cite

As the demand for real-time data processing increases, a high-speed processing platform for large-scale stream data becomes necessary. For fast processing large-scale stream data, it is essential to use multiple distributed nodes. So far, there have been few studies on real-time massive image processing through efficient management and allocation of heterogeneous resources for various user-specified nodes on distributed environments. In this paper, we shall present a new platform called RIDE (Real-time massive Image processing platform on Distributed Environment) which efficiently allocates resources and executes load balancing according to the amount of stream data on distributed environments. It minimizes communication overhead by using a parallel processing strategy which handles the stream data considering both coarse-grained and fine-grained parallelism simultaneously. Coarse-grained parallelism is achieved by the automatic allocation of input streams onto partitions of broker buffer each processed by its corresponding worker node, and maximized by adaptive resource management which adjusts the number of worker nodes in a group according to the frame rate in real time. Fine-grained parallelism is achieved by parallel processing of task on each worker node and maximized by allocating heterogeneous resources such as GPU and embedded machines appropriately. Moreover, it provides a scheme of application topology which has a great advantage for higher performance by configuring the worker nodes of each stage using adaptive heterogeneous resource management. Finally, it supports dynamic fault tolerance for real-time image processing through the coordination between components in our system.

show abstract

A survey on parallel computing for traditional computer vision

Jaiswal

Kumar

2021

Concurrency and Computation

View full text Add to dashboard Cite

Summary The applications of computer vision (CV) are continuously increasing along with the enormous demand for real‐time data processing. This visual data processing is done with various compute‐intensive image/video processing algorithms that may belong to traditional approaches or deep learning approaches. This article aims to provide a survey of state‐of‐the‐art hardware platforms and software frameworks for parallel implementation of traditional CV applications. The article discusses various options for hardware platforms for centralized‐computing architecture and edge‐computing architecture, and various software frameworks that can be used to leverage the hardware. This discussion is based on a systematic survey of studies/works that show the use of various hardware platforms and software frameworks in order to achieve real‐time processing for CV algorithms. Based on the survey, some possible future directions are also discussed.

show abstract

Large Scale Image Processing in Real-Time Environments with Kafka

Abstract: Recently, real-time image data generated is increasing not only in resolution but

Cited by 6 publications

References 2 publications

Exploring Real-Time Sentiment Analysis Prototype for Retail Industry

Exploring Real-Time Sentiment Analysis Prototype for Retail Industry

RIDE: real-time massive image processing platform on distributed environment

A survey on parallel computing for traditional computer vision

Contact Info

Product

Resources

About