Implementing Big Data Lake for Heterogeneous Data Sources

Mehmood, Hassan; Gilman, Ekaterina; Cortés, Marta; Kostakos, Panos; Byrne, Andrew; Valta, Katerina; Tekes, Stavros; Riekki, Jukka

doi:10.1109/icdew.2019.00-37

Cited by 55 publications

(67 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Such developments not only make cities smarter, but also help in improving life quality of its inhabitants for example, well-being through technology, planning routine with smart applications, e-learning and others. While the concept of the "smart city" still remains elusive, modern cities are increasingly becoming dependant on a symbiotic relationship with proliferating technologies such as, IoT, sensors, distributed cloud, fog/edge computing, smartphones, and others [18,19]. According to recent estimations, the market for solutions and services created for smart cities around the globe will approximately grow over 2 trillion USD [20].…”

Section: Overview On Advances In Smart Citiesmentioning

confidence: 99%

Concept Drift Adaptation Techniques in Distributed Environment for Real-World Data Streams

et al. 2021

Self Cite

View full text Add to dashboard Cite

Real-world data streams pose a unique challenge to the implementation of machine learning (ML) models and data analysis. A notable problem that has been introduced by the growth of Internet of Things (IoT) deployments across the smart city ecosystem is that the statistical properties of data streams can change over time, resulting in poor prediction performance and ineffective decisions. While concept drift detection methods aim to patch this problem, emerging communication and sensing technologies are generating a massive amount of data, requiring distributed environments to perform computation tasks across smart city administrative domains. In this article, we implement and test a number of state-of-the-art active concept drift detection algorithms for time series analysis within a distributed environment. We use real-world data streams and provide critical analysis of results retrieved. The challenges of implementing concept drift adaptation algorithms, along with their applications in smart cities, are also discussed.

show abstract

Section: Overview On Advances In Smart Citiesmentioning

confidence: 99%

Concept Drift Adaptation Techniques in Distributed Environment for Real-World Data Streams

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…The highlighted challenges of city-scale computing can coarsely be summarised as (1) large-scale data quantity and how to process it efficiently [3], (2) heterogeneous data providers in terms of data quality, source (varying from private citizens to vehicles and industrial applications), and sampling frequency [6,7], (3) mobile and low-capacity devices as a part of the system, especially private carry-on devices and vehicles [10,27], and (4) real-time requirements to produce efficient recommendations, situational awareness, and other big-picture services and applications [10]. Some platforms are suggested for city-scale computation activities, including both "traditional" elastic cloud services [28] and data lakes [29].…”

Section: Related Workmentioning

confidence: 99%

EDISON: An Edge-Native Method and Architecture for Distributed Interpolation

Lovén

Lähderanta

Ruha

et al. 2021

Sensors

Self Cite

View full text Add to dashboard Cite

Spatio-temporal interpolation provides estimates of observations in unobserved locations and time slots. In smart cities, interpolation helps to provide a fine-grained contextual and situational understanding of the urban environment, in terms of both short-term (e.g., weather, air quality, traffic) or long term (e.g., crime, demographics) spatio-temporal phenomena. Various initiatives improve spatio-temporal interpolation results by including additional data sources such as vehicle-fitted sensors, mobile phones, or micro weather stations of, for example, smart homes. However, the underlying computing paradigm in such initiatives is predominantly centralized, with all data collected and analyzed in the cloud. This solution is not scalable, as when the spatial and temporal density of sensor data grows, the required transmission bandwidth and computational capacity become unfeasible. To address the scaling problem, we propose EDISON: algorithms for distributed learning and inference, and an edge-native architecture for distributing spatio-temporal interpolation models, their computations, and the observed data vertically and horizontally between device, edge and cloud layers. We demonstrate EDISON functionality in a controlled, simulated spatio-temporal setup with 1 M artificial data points. While the main motivation of EDISON is the distribution of the heavy computations, the results show that EDISON also provides an improvement over alternative approaches, reaching at best a 10% smaller RMSE than a global interpolation and 6% smaller RMSE than a baseline distributed approach.

show abstract

“…Storage Layer: Our platform's secure large scale storage and processing infrastructure is based on a hybrid cloud based approach (Mehmood et al, 2019), with a flexible and horizontally scalable architecture that can be easily customized to meet the specific needs of different public administrations and uses open source technologies such as Kafka (Kafka, 2020), Elasticsearch (Elasticsearch, 2020), and Kibana (Kibana, 2020) for data management and processing. The solution leverages open source and proprietary technologies and takes a security-by-design approach to address significant threats (CUTLER, 2020d).…”

Section: Architecturementioning

confidence: 99%

Evidence-driven policy-making using heterogeneous data sources—The case of a controlled parking system in Thessaloniki

et al. 2020

View full text Add to dashboard Cite

Policy-making in local public administrations is still largely based on intuition rather than being backed up by data and evidence. The goal of this work is to introduce the methodology and software tools for contributing toward transforming the existing intuition-based paradigm of policy-making into an evidence-driven approach enabled by heterogeneous sources of data already available in the city. More specifically, methods for data collection, efficient data storage, and data analysis are implemented to measure the economic activity, assess the environmental impact and evaluate the social consequences of certain policy decisions. Subsequently, the extracted pieces of evidence are used to inform, advise, monitor, evaluate, and revise the decisions made by policy planners. Our contribution in this work is on outlining and deploying an easily extendable system architecture to harmonize and analyze heterogeneous data sources in ways that are found to be useful for policy-makers. For evaluating this architecture, we examine the case of a controlled parking system in the city of Thessaloniki and try to optimize its operation by balancing effectively between economic growth, environmental protection, and citizen satisfaction.

show abstract

Implementing Big Data Lake for Heterogeneous Data Sources

Cited by 55 publications

References 23 publications

Concept Drift Adaptation Techniques in Distributed Environment for Real-World Data Streams

Concept Drift Adaptation Techniques in Distributed Environment for Real-World Data Streams

EDISON: An Edge-Native Method and Architecture for Distributed Interpolation

Evidence-driven policy-making using heterogeneous data sources—The case of a controlled parking system in Thessaloniki

Contact Info

Product

Resources

About