An Integrated Big and Fast Data Analytics Platform for Smart Urban Transportation Management

Fiore, Sandro; Elia, Donatello; Pires, Carlos Eduardo Santos; Mestre, Demetrio Gomes; Cappiello, Cinzia; Vitali, Monica; Andrade, Nazareno; Braz, Tarciso; Lezzi, Daniele; Moraes, Regina; Basso, Tânia; Kozievitch, Nádia P.; Fonseca, Keiko V. O.; Antunes, Nuno; Vieira, Marco; Palazzo, Cosimo; Blanquer, Ignácio; Meira, Wagner; Aloisio, Giovanni

doi:10.1109/access.2019.2936941

Cited by 46 publications

(18 citation statements)

References 62 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Ophidia framework represents an open source solution 5 for scientific data analytics, joining HPC paradigms and Big Data approaches. The framework has primarily been used in the climate change domain, though it has also successfully been exploited in other domains/contexts (e.g., astronomy, seismology [88], smart cities [89]). Ophidia addresses scientific data analysis on large multi-dimensional datasets, leveraging the datacube abstraction inherited from the OLAP data warehouse systems.…”

Section: The Ophidia Hpda Frameworkmentioning

confidence: 99%

Towards HPC and Big Data Analytics Convergence: Design and Experimental Evaluation of a HPDA Framework for eScience at Scale

2021

Self Cite

View full text Add to dashboard Cite

Over the last two decades, scientific discovery has increasingly been driven by the large availability of data from a multitude of sources, including high-resolution simulations, observations and instruments, as well as an enormous network of sensors and edge components. In such a dynamic and growing landscape where data continue to expand, advances in Science have become intertwined with the capacity of analysis tools to effectively handle and extract valuable information from this ocean of data. In view of the exascale era of supercomputers that is rapidly approaching, it is of the utmost importance to design novel solutions that can take full advantage of the upcoming computing infrastructures. The convergence of High Performance Computing (HPC) and data-intensive analytics is key to delivering scalable High Performance Data Analytics (HPDA) solutions for scientific and engineering applications. The aim of this paper is threefold: reviewing some of the most relevant challenges towards HPDA at scale, presenting a HPDAenabled version of the Ophidia framework and validating the scalability of the proposed framework through an experimental performance evaluation carried out in the context of the Centre of Excellence in Simulation of Weather and Climate in Europe (ESiWACE). The experimental results show that the proposed solution is capable of scaling over several thousand cores and hundreds of cluster nodes. The proposed work is a contribution in support of scientific large-scale applications along the wider convergence path of HPC and Big Data followed by the scientific research community.

show abstract

Section: The Ophidia Hpda Frameworkmentioning

confidence: 99%

Towards HPC and Big Data Analytics Convergence: Design and Experimental Evaluation of a HPDA Framework for eScience at Scale

2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…All the data used in this study were collected for the period of July, August and September 2019 (Only weekdays). Integrating and aggregating these data sets to evaluate a smart transport system is one of the main challenges that transport researchers and agencies currently face [81]. These data sets can be very useful information for other authors and researcher to conduct further studies.…”

Section: Datamentioning

confidence: 99%

Using simulation model as a tool for analyzing bus service reliability and implementing improvement strategies

2020

View full text Add to dashboard Cite

Bus services naturally tend to be unstable and are not always capable of adhering to schedules without control strategies. Therefore, bus users and bus service providers face travel time variation and irregularity. After a comprehensive review of the literature, a significant gap was recognized in the field of public transportation reliability. According to literature, there is no consistency in reliability definition and indicators. Companies have their own definition of bus service reliability, and they mostly neglect the passengers' perspective of reliability. Therefore, four reliability indicators were selected in this study to fill the gap in the literature and cover both passengers' and operators' perceptions of reliability: waiting time and on-board crowding level from passengers' perspective, and headway regularity index at stops (HRIS) and bus bunching/big gap percentage from operators' perspective. The primary objective of this research is to improve the reliability of high frequency of bus service and simulation tools currently being used by the public transportation companies. Therefore, a simulation model of bus service was developed to study the strategies to alleviate it. Four different types of strategies were selected and implemented according to Route U32 (Kuala Lumpur) specifications. Model output showed that control strategies such as headwaybased dispatching could significantly improve headway regularity by almost 62% and the waiting time by 51% on average. Both holding strategies at key stops (previous and Prefol holding) have shown an almost similar impact on reliability indicators. Waiting time was reduced by 44% and 43% after the previous and Prefol Headway strategies were adopted, respectively. However, the implementation of the component of headway-based strategies at the terminal and key stops showed the best impact on reliability, in terms of passenger waiting time. Waiting time and excess waiting time were both significantly reduced by 52.86% and 81.44%, respectively. Nevertheless, the strategies did not show any significant positive effect on the level of crowding during morning peak hours.

show abstract

“…end if (7) node � node, Node (codeY (i)) (8) end for (9) node.addFile (fileIndex) ALGORITHM 2: AttributeInsert. Complexity of the data is stored as values in key-value pairs.…”

Section: Integration and Urban Applicationmentioning

confidence: 99%

“…erefore, from the aspect of big data management, the raw urban data are massive, distributed, heterogeneous, and inconsistent. According to the needs of smart city services, urban data generated from different sources need to be consolidated for storage, retrieval, and analysis [7]. In this area, the latest urban data management research has proposed many specific integration methods and architectures on the urban data [8,9,10].…”

Section: Introductionmentioning

confidence: 99%

Solving the Fragment Complexity of Official, Social, and Sensorial Urban Data

et al. 2020

View full text Add to dashboard Cite

Cities in the big data era hold the massive urban data to create valuable information and digitally enhanced services. Sources of urban data are generally categorized as one of the three types: official, social, and sensorial, which are from the government and enterprises, social networks of citizens, and the sensor network. These types typically differ significantly from each other but are consolidated together for the smart urban services. Based on the sophisticated consolidation approaches, we argue that a new challenge, fragment complexity that represents a well-integrated data has appropriate but fragmentary schema and difficult to be queried, is ignored in the state-of-art urban data management. Comparing with predefined and rigid schema, fragmentary schema means a dataset contains millions of attributes but nonorthogonally distributed among tables, and of course, values of these attributes are even massive. As far as a query is concerned, locating where these attributes are being stored is the first encountered problem, while traditional value-based query optimization has no contributions. To address this problem, we propose an index on massive attributes as an attributes-oriented optimization, namely, attribute index. Attribute index is a secondary index for locating files in which the target attributes are stored. It contains three parts: ATree for searching keys, DTree for locating keys among files, and ADLinks as a mapping table between ATree and DTree. In this paper, the index architecture, logical structure and algorithms, the implementation details, the creation process, the integration to the existing key-value store, and the urban application scenario are described. Experiments show that, in comparison with B + -Tree, LSM-Tree, and AVL-Tree, the query time of ATree is 1.1x, 1.5x, and 1.2x faster, respectively. Finally, we integrate our proposition with HBase, namely, UrbanBase, whose query performance is 1.3x faster than the original HBase.

show abstract

An Integrated Big and Fast Data Analytics Platform for Smart Urban Transportation Management

Cited by 46 publications

References 62 publications

Towards HPC and Big Data Analytics Convergence: Design and Experimental Evaluation of a HPDA Framework for eScience at Scale

Towards HPC and Big Data Analytics Convergence: Design and Experimental Evaluation of a HPDA Framework for eScience at Scale

Using simulation model as a tool for analyzing bus service reliability and implementing improvement strategies

Solving the Fragment Complexity of Official, Social, and Sensorial Urban Data

Contact Info

Product

Resources

About