Big Data Management: What to Keep from the Past to Face Future Challenges?

Abstract

The emergence of new hardware architectures, and the continuous production of data open new challenges for data management. It is no longer pertinent to reason with respect to a predefined set of resources (i.e., computing, storage and main memory). Instead, it is necessary to design data processing algorithms and processes considering unlimited resources via the ''pay-as-you-go'' model. According to this model, resources provision must consider the economic cost of the processes versus the use and parallel exploitation of available computing resources. In consequence, new methodologies, algorithms and tools for querying, deploying and programming data management functions have to be provided in scalable and elastic architectures that can cope with the characteristics of Big Data aware systems (intelligent systems, decision making, virtual environments, smart cities, drug personalization). These functions, must respect QoS properties (e.g., security, reliability, fault tolerance, dynamic evolution and adaptability) and behavior properties (e.g., transactional execution) according to application requirements. Mature and novel system architectures propose models and mechanisms for adding these properties to new efficient data management and processing functions delivered as services. This paper gives an overview of the different architectures in which efficient data management functions can be delivered for addressing Big Data processing challenges.

Accurately Estimating User Cardinalities and Detecting Super Spreaders Over Time

“…Monitoring the cardinalities (or degrees) of users in these networks is fundamental for many applications such as network anomaly detection [15], [54], where a user's cardinality is defined to be the number of distinct users/items that the user connects to in the unipartite/bipartite graph stream of interest. Due to the large-size and high-speed nature of these graph streams, it is infeasible to collect the entire graph especially when the computational and memory resources are limited [44], [37]. For example, network routers have fast but very small memories, which leads their traffic monitoring modules incapable to exactly compute the cardinalities of network users.…”

Distributed Similarity Queries in Metric Spaces

“…With the development of Internet, especially the widespread use of mobile devices, distributed data processing becomes a booming area in both the data management industry and academia [1][2][3][4][5]. Nowadays, the volume, richness and diversity of data challenges traditional metric similarity query processing in both space and time.…”

GREENHOME: A Household Energy Consumption and CO 2 Footprint Metering Environment

“…Load profiling. Used to determine basic electricity consumption patterns of different costumers' groups by classifying consumers' load curves according to their energy consumption behaviour: (i) direct clustering-based approach with different classification techniques used like K-means [20] [42], hierarchical clustering [61], and self-organising map (SOM) [50]; (ii) indirect clustering includes dimensionality reduction, load characteristics and uncertainty-based methods depending on the features extracted before clustering.…”