Real-Time Traffic Monitoring System Using Spark

Saraswathi, A; Mummoorthy, A.; R, Anantha Raman G; Porkodi, K P

doi:10.1109/icese46178.2019.9194613

Cited by 8 publications

(3 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Mishra et al [31] proposed a framework to predict congestion on multivariate IoT data streams in a smart city scenario, using Apache Spark to receive and process data from Apache Kafka. A. Saraswathi et al [32] also used Kafka and Spark to predict road traffic in real-time. Y. Drohobytskiy et al [33] developed a real-time multi-party data exchange using Apache Spark to obtain data from Apache Kafka, process it and store it in HDFS.…”

Section: Data Processmentioning

confidence: 99%

Parameterization and Performance Analysis of a Scalable, near Real-Time Packet Capturing Platform

Oliveira,

Pedrosa,

Rufino

et al. 2024

Systems

View full text Add to dashboard Cite

The rapid evolution of technology has fostered an exponential rise in the number of individuals and devices interconnected via the Internet. This interconnectedness has prompted companies to expand their computing and communication infrastructures significantly to accommodate the escalating demands. However, this proliferation of connectivity has also opened new avenues for cyber threats, emphasizing the critical need for Intrusion Detection Systems (IDSs) to adapt and operate efficiently in this evolving landscape. In response, companies are increasingly seeking IDSs characterized by horizontal, modular, and elastic attributes, capable of dynamically scaling with the fluctuating volume of network data flows deemed essential for effective monitoring and threat detection. Yet, the task extends beyond mere data capture and storage; robust IDSs must integrate sophisticated components for data analysis and anomaly detection, ideally functioning in real-time or near real-time. While Machine Learning (ML) techniques present promising avenues for detecting and mitigating malicious activities, their efficacy hinges on the availability of high-quality training datasets, which in turn poses a significant challenge. This paper proposes a comprehensive solution in the form of an architecture and reference implementation for (near) real-time capture, storage, and analysis of network data within a 1 Gbps network environment. Performance benchmarks provided offer valuable insights for prototype optimization, demonstrating the capability of the proposed IDS architecture to meet objectives even under realistic operational scenarios.

show abstract

Section: Data Processmentioning

confidence: 99%

Parameterization and Performance Analysis of a Scalable, near Real-Time Packet Capturing Platform

Oliveira,

Pedrosa,

Rufino

et al. 2024

Systems

View full text Add to dashboard Cite

show abstract

“…Gaur et al [39] used Apache Spark big data as a processing tool over smarthphone accelerometere datasets. Saraswathi et al [40] used Apache Spark as big data engine for real time traffic monitoring system to predict the total traffic count of streaming data in various routes to reduce traffic congestion. IoT applications face unique problems related to their widely distributed, resource-constrained device endpoints [41].…”

Section: (B)mentioning

confidence: 99%

Big Data in Smart City: Management Challenges

et al. 2021

View full text Add to dashboard Cite

Smart cities use digital technologies such as cloud computing, Internet of Things, or open data in order to overcome limitations of traditional representation and exchange of geospatial data. This concept ensures a significant increase in the use of data to establish new services that contribute to better sustainable development and monitoring of all phenomena that occur in urban areas. The use of the modern geoinformation technologies, such as sensors for collecting different geospatial and related data, requires adequate storage options for further data analysis. In this paper, we suggest the biG dAta sMart cIty maNagEment SyStem (GAMINESS) that is based on the Apache Spark big data framework. The model of the GAMINESS management system is based on the principles of the big data modeling, which differs greatly from standard databases. This approach provides the ability to store and manage huge amounts of structured, semi-structured, and unstructured data in real time. System performance is increasing to a higher level by using the process parallelization explained through the five V principles of the big data paradigm. The existing solutions based on the five V principles are focused only on the data visualization, not the data themselves. Such solutions are often limited by different storage mechanisms and by the ability to perform complex analyses on large amounts of data with expected performance. The GAMINESS management system overcomes these disadvantages by conversion of smart city data to a big data structure without limitations related to data formats or use standards. The suggested model contains two components: a geospatial component and a sensor component that are based on the CityGML and the SensorThings standards. The developed model has the ability to exchange data regardless of the used standard or the data format into proposed Apache Spark data framework schema. The verification of the proposed model is done within the case study for the part of the city of Novi Sad.

show abstract

“…It has received considerable attention as an alternative approach to overcoming these limitations. Apache Spark has been extensively implemented in numerous research fields and applications, such as healthcare systems [19], and traffic [20,21]. The integration of ensemble methods and efficient real-time data processing enhances the capability of IDSs to detect and classify common attacks in the CAN bus network including Denial-of-Service (DoS), fuzzing, replay, and spoofing.…”

Section: Introductionmentioning

confidence: 99%

An Effective Ensemble Learning-Based Real-Time Intrusion Detection Scheme for an In-Vehicle Network

Alalwany,

Mahgoub

2024

Electronics

View full text Add to dashboard Cite

The emergence of connected and autonomous vehicles has led to complex network architectures for electronic control unit (ECU) communication. The controller area network (CAN) enables the transmission of data inside vehicle networks. However, although it has low latency and enjoys data broadcast capability, it is vulnerable to attacks on security. The lack of effectiveness of conventional security mechanisms in addressing these vulnerabilities poses a danger to vehicle safety. This study presents an intrusion detection system (IDS) that accurately detects and classifies CAN bus attacks in real-time using ensemble techniques and the Kappa Architecture. The Kappa Architecture enables real-time attack detection, while ensemble learning combines multiple machine learning classifiers to enhance the accuracy of attack detection. The scheme utilizes ensemble methods with Kappa Architecture’s real-time data analysis to detect common CAN bus attacks. This study entails the development and evaluation of supervised models, which are further enhanced using ensemble techniques. The accuracy, precision, recall, and F1 score are used to measure the scheme’s effectiveness. The stacking ensemble technique outperformed individual supervised models and other ensembles with accuracy, precision, recall, and F1 of 0.985, 0.987, and 0.985, respectively.

show abstract

Real-Time Traffic Monitoring System Using Spark

Cited by 8 publications

References 9 publications

Parameterization and Performance Analysis of a Scalable, near Real-Time Packet Capturing Platform

Parameterization and Performance Analysis of a Scalable, near Real-Time Packet Capturing Platform

Big Data in Smart City: Management Challenges

An Effective Ensemble Learning-Based Real-Time Intrusion Detection Scheme for an In-Vehicle Network

Contact Info

Product

Resources

About