Programming big data analysis: principles and solutions

Belcastro, Loris; Cantini, Riccardo; Marozzo, Fabrizio; Orsino, Alessio; Talia, Domenico; Trunfio, Paolo

doi:10.1186/s40537-021-00555-2

Cited by 40 publications

(14 citation statements)

References 63 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Figure 6 shows the MapReduce data flow, a model proposed by Google, which addresses a new programming paradigm for working with Big Data. This model allows the manipulation of Big Data in parallel and distributed way, in addition to providing fault tolerance, scaling Input/Output (I/O), and monitoring [18][19][20][21][22][23].…”

Section: The Seasonal Climate Predictionmentioning

confidence: 99%

“…6 The MapReduce Data Flow [18] down into lines; (3) Mapping-when each part (line) is computed for the key/value format; (4) Sort/Shuffle-when the Sort and Shuffle operations perform the sorting and grouping of data, according to the "key"; (5) Reducing-when calculating the values contained in each grouping; and, finally, (6) Output-when the result is recorded on the HDFS. Figure 7 shows the MapReduce flow applied to count words in a text [18][19][20][21][22][23].…”

Section: The Seasonal Climate Predictionmentioning

confidence: 99%

See 1 more Smart Citation

A canonical model for seasonal climate prediction using Big Data

et al. 2022

View full text Add to dashboard Cite

This article addresses the elaboration of a canonical model, involving methods, techniques, metrics, tools, and Big Data, applied to the knowledge of seasonal climate prediction, aiming at greater dynamics, speed, conciseness, and scalability. The proposed model was hosted in an environment capable of integrating different types of meteorological data and centralizing data stores. The seasonal climate prediction method called M-PRECLIS was designed and developed for practical application. The usability and efficiency of the proposed model was tested through a case study that made use of operational data generated by an atmospheric numerical model of the climate area found in the supercomputing environment of the Center for Weather Forecasting and Climate Studies linked to the Brazilian Institute for Space Research. The seasonal climate prediction uses ensemble members method to work and the main Big Data technologies used for data processing were: Python language, Apache Hadoop, Apache Hive, and the Optimized Row Columnar (ORC) file format. The main contributions of this research are the canonical model, its modules and internal components, the proposed method M-PRECLIS, and its use in a case study. After applying the model to a practical and real experiment, it was possible to analyze the results obtained and verify: the consistency of the model by the output images, the code complexity, the performance, and also to perform the comparison with related works. Thus, it was found that the proposed canonical model, based on the best practices of Big Data, is a viable alternative that can guide new paths to be followed.

show abstract

Section: The Seasonal Climate Predictionmentioning

confidence: 99%

Section: The Seasonal Climate Predictionmentioning

confidence: 99%

A canonical model for seasonal climate prediction using Big Data

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Specifically, the sum of the membership, m S (a i ), and non-membership, m S *(a i ), is not necessarily one, then: 0 ≤ m S (a i ) + m* S (a i ) ≤ 1. Additionally, the hesitation h S (a i ) h S (a i ) = 1 − (m S (a i ) + m* S (a i )) (6) is the degree of indeterminacy (hesitation).…”

Section: Intuitionistic Fuzzy Setsmentioning

confidence: 99%

“…This can include sensor information and data ranging to the subjective interpretations obtained from expert individuals and analysts. Currently, increasingly massive amounts of heterogeneous data and information from multiple sources are prevalent where the problems of Big Data are being managed [4][5][6][7]. However, although effective decision making should be able to make use of all the available and relevant information about such combined uncertainty, an assessment of the value of a generalization result is critical.…”

Section: Introductionmentioning

confidence: 99%

Intuitionistic and Interval-Valued Fuzzy Set Representations for Data Mining

Petry

Yager

2022

Algorithms

View full text Add to dashboard Cite

Data mining refers to a variety of techniques in the fields of databases, machine learning and pattern recognition. The intent is to obtain useful patterns and associations from a large collection of data. In this paper we describe extensions to the attribute generalization process to deal with interval and intuitionistic fuzzy information. Specifically, we consider extensions for using interval-valued fuzzy representations in both data and the generalization hierarchy. Moreover, preliminary representations using intuitionistic fuzzy information for attribute generalization are described. Finally, we consider how to use fuzzy hierarchies for the generalization of interval-valued fuzzy representations.

show abstract

“…The rapid spread of Internet of Things (IoT) devices is generating huge volumes of data at the network edge [1]. Managing this data flow using highly centralized solutions, such as those based on cloud platforms, is extremely ineffective in terms of response time, network traffic management, power consumption, and scalability [2].…”

Section: Introductionmentioning

confidence: 99%

Edge-Cloud Continuum Solutions for Urban Mobility Prediction and Planning

et al. 2023

Self Cite

View full text Add to dashboard Cite

In recent years, there has been an increase in the use of edge-cloud continuum solutions to efficiently collect and analyze data generated by IoT devices. In this paper, we investigate to what extent these solutions can manage tasks related to urban mobility, by combining real-time and low latency analysis offered by the edge with large computing and storage resources provided by the cloud. Our proposal is organized into three parts. The first part focuses on defining three application scenarios in which geotagged data generated by IoT objects, such as taxis, cars, and smartphones, are collected and analyzed through machine learning algorithms (i.e., next location prediction, location-based advertising, and points of interest recommendation). The second part is dedicated to modeling an edge-cloud continuum architecture capable of managing a large number of IoT devices and executing machine learning algorithms to analyze the data they generate. The third part analyzes the experimental results in which different design choices were evaluated, such as the number of devices and orchestration policies, to improve the performance of machine learning algorithms in terms of processing time, network delay, task failure, and computational resource utilization. The results highlight the potential benefits of edge and cloud cooperation in the three application scenarios, demonstrating that it significantly improves resource utilization and reduces the task failure rate compared to other widely adopted architectures, such as edge-or cloud-only architectures.INDEX TERMS Edge-cloud architecture, IoT infrastructure, Edge computing, Urban computing, Smart cities, Urban mobility

show abstract

Programming big data analysis: principles and solutions

Cited by 40 publications

References 63 publications

A canonical model for seasonal climate prediction using Big Data

A canonical model for seasonal climate prediction using Big Data

Intuitionistic and Interval-Valued Fuzzy Set Representations for Data Mining

Edge-Cloud Continuum Solutions for Urban Mobility Prediction and Planning

Contact Info

Product

Resources

About