A Distributed Big Data Analytics Architecture for Vehicle Sensor Data

Abstract

The unceasingly increasing needs for data acquisition, storage and analysis in transportation systems have led to the adoption of new technologies and methods in order to provide efficient and reliable solutions. Both highways and vehicles, nowadays, host a vast variety of sensors collecting different types of highly fluctuating data such as speed, acceleration, direction, and so on. From the vast volume and variety of these data emerges the need for the employment of big data techniques and analytics in the context of state-of-the-art intelligent transportation systems (ITS). Moreover, the scalability needs of fleet and traffic management systems point to the direction of designing and deploying distributed architecture solutions that can be expanded in order to avoid technological and/or technical entrapments. Based on the needs and gaps detected in the literature as well as the available technologies for data gathering, storage and analysis for ITS, the aim of this study is to provide a distributed architecture platform to address these deficiencies. The architectural design of the system proposed, engages big data frameworks and tools (e.g., NoSQL Mongo DB, Apache Hadoop, etc.) as well as analytics tools (e.g., Apache Spark). The main contribution of this study is the introduction of a holistic platform that can be used for the needs of the ITS domain offering continuous collection, storage and data analysis capabilities. To achieve that, different modules of state-of-the-art methods and tools were utilized and combined in a unified platform that supports the entire cycle of data acquisition, storage and analysis in a single point. This leads to a complete solution for ITS applications which lifts the limitations imposed in legacy and current systems by the vast amounts of rapidly changing data, while offering a reliable system for acquisition, storage as well as timely analysis and reporting capabilities of these data.

Lane-Level Map Generation and Management Framework Using Connected Car Data

“…They used this schema to build a distributed data processing platform that allows the detection of abnormal data from vehicles. Theodoros et al [22] proposed a system that introduces a distributed architecture platform for the continuous collection, storage, and analysis of vehicle state data. Meanwhile, David et al [23] collected real-time road environment data through a vehicle's onboard diagnostics II port and smartphone devices.…”

Network Resources Optimization through Regional Computing forVehicular Big Data

“…Likewise, in [23], authors propose a big data analytical architecture tailored for vehicular data. Their approach advocates for a distributed architecture accommodating data processing, storage, and analysis to effectively handle the immense data volumes.…”

Dynamic Anomaly Detection in Gantry Transactions Using Graph Convolutional Network-Gate Recurrent Unit with Adaptive Attention

“…However, since the system involves a large number of devices and data transmission, the occurrence of system failures may lead to problems such as traffic congestion and loss of fees. Therefore, the efficient dynamic anomaly detection of ETC gantry can help maintenance personnel to quickly detect and solve system anomalies, ensuring the normal operation of the ETC gantry system, which is of great significance, and can also provide a data basis and guarantee for a series of research [4][5][6][7] on expanding the application of smart highways based on ETC transaction data. The original ETC gantry anomaly detection mainly uses manual inspection or remote detection.…”