A Machine Learning and Multi-Agent Model to Automate Big Data Analytics in Smart Cities

Abstract

The objective of this paper is to present an architecture to improve the process of automating big data analytics using a multi-agent system and machine learning techniques, to support the processing of real time big data streams and to enhance the process of decision-making for urban planning and management. With the rapidly evolving information technologies, and their utilization in many areas such as smart cities, social networks, urban management and planning, massive data streams are generated and need an efficient approach to deal with. The proposition in this paper adopts the concept of smart data which focuses on the value aspect from big data. The proposed architecture is composed of three layers: data acquisition and storage, data management and processing and the service layer, based on a multi-agent system to automate the big data analytics; the proposed model describe the functionalities of the system and the collaboration between agents, these autonomous entities receive data streams in real time, they perform operations of preprocessing, big data analytics and storage into a Hadoop cluster. The techniques of machine learning are also used to enhance the process of decision making, such the use of classification algorithms to predict habitat type based on the characteristics of a population to help making efficient urban planning decisions. The proposed system can serve as a platform to support data management and to conduct effective decision-making in smart cities.

Navigating the Ethical and Privacy Concerns of Big Data and Machine Learning in Decision Making

“…However, there are also challenges associated with the use of machine learning and big data, as highlighted in the studies by Ning and You [77] and Wang et al [78] . These articles emphasize the need to consider uncertainty and potential biases in the development and implementation of these technologies.…”

“…Despite obstacles, there are numerous opportunities to utilize big data and machine learning in a variety of fields. For instance, machine learning models can be utilized in sports to predict athlete performance and assess influencing factors [78] . Online incremental machine learning platforms can be developed for big data-driven intelligent traffic management in the transportation sector [67] .…”

Towards a Machine Learning-based Model for Corporate Loan Default Prediction

“…As previously cited, the following algorithms are performed: To train the prepared dataset, the dataset is split into a training set of 80% and 20% for testing as used for training classification machine learning algorithms [44].…”