An efficient parallel indexing structure for multi-dimensional big data using spark

Elmeiligy, Manar A.; Desouky, Ali I. El; Elghamrawy, Sally M.

doi:10.1007/s11227-021-03718-3

Cited by 7 publications

(4 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This method also diminishes the cold start problem of collaborative filtering by correlating the users to products through features (tags). Manar et al [31] have proposed a Comprehensive Storing System for COVID-19 data using Apache Spark (CSS-COVID) to address the problem caused by increasing the number of COVID-19 daily. CSS-COVID consists of three stages, namely, inserting and indexing, storing, and querying stage.…”

Section: Apache Spark Applicationsmentioning

confidence: 99%

A distributed computing model for big data anonymization in the networks

Ashkouti¹,

Khamforoosh²

2023

PLoS ONE

View full text Add to dashboard Cite

Recently big data and its applications had sharp growth in various fields such as IoT, bioinformatics, eCommerce, and social media. The huge volume of data incurred enormous challenges to the architecture, infrastructure, and computing capacity of IT systems. Therefore, the compelling need of the scientific and industrial community is large-scale and robust computing systems. Since one of the characteristics of big data is value, data should be published for analysts to extract useful patterns from them. However, data publishing may lead to the disclosure of individuals’ private information. Among the modern parallel computing platforms, Apache Spark is a fast and in-memory computing framework for large-scale data processing that provides high scalability by introducing the resilient distributed dataset (RDDs). In terms of performance, Due to in-memory computations, it is 100 times faster than Hadoop. Therefore, Apache Spark is one of the essential frameworks to implement distributed methods for privacy-preserving in big data publishing (PPBDP). This paper uses the RDD programming of Apache Spark to propose an efficient parallel implementation of a new computing model for big data anonymization. This computing model has three-phase of in-memory computations to address the runtime, scalability, and performance of large-scale data anonymization. The model supports partition-based data clustering algorithms to preserve the λ-diversity privacy model by using transformation and actions on RDDs. Therefore, the authors have investigated Spark-based implementation for preserving the λ-diversity privacy model by two designed City block and Pearson distance functions. The results of the paper provide a comprehensive guideline allowing the researchers to apply Apache Spark in their own researches.

show abstract

Section: Apache Spark Applicationsmentioning

confidence: 99%

A distributed computing model for big data anonymization in the networks

Ashkouti¹,

Khamforoosh²

2023

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…The developed approach employs a user interface that offers multiple inputs to the search engine. Elmeiligy et al (2021) proposed a new parallel indexing system based on spark (ParISSS). The proposed ParISSS is mainly used for the analysis of multidimensional big data.…”

Section: Related Workmentioning

confidence: 99%

“…features are between 1 and 15 in this assessment. The methods of hierarchical dynamic covering with cross-range constraints (HDCRC) (Sun et al, 2020), parallel indexing system based on spark (ParISSS) (Elmeiligy et al, 2021), and latent semantic indexing (LSI) for recommender systems (RS) (LSI-CRS) (Horasan, 2022) are accounted for from the Related Works section in this comparative study.…”

Section: Performance Assessmentmentioning

confidence: 99%

Research on an evaluation index system of critical emergency management capability based on machine learning in a complex scientific environment

Zhao

2023

Front. Ecol. Evol.

View full text Add to dashboard Cite

IntroductionA complex scientific environment requires multiple considerations for handling critical and emergency conditions with an addressing solution. Indexing and prioritizing are standard methods that are used in such settings to improve itinerary solutions. Significance of an indexing system relies on the benchmark solution and the strategy it implies.MethodsThe present study introduces an indexing strategy evaluation method (ISEM) to validate the efficiency of indexing systems. The proposed method identifies the root implication and the strategy parameters to address complex problems. The environmental and problem-specific parameters are determined to estimate the system's initial response. The capability through solution response, lag, and failure analysis is identified post the estimation through linear regression learning. The indexing system's operations are designed through linear itineraries to prevent interrupting failures. In addition, the environmental features are identified as augmenting factors to prevent strategy pausing across multiple indices.Results and discussionThe proposed method employs linear analysis through itinerary levels of index evaluation for optimal, lagging, and failed implications. It also helps to identify specific reasons for solution improvement or retention from previous operations.

show abstract

“…Technological frameworks like Apache Spark have been determinant for the widespread adoption of cluster-based systems in many areas, including machine learning. Recent advances show promising implementations of SAM on Apache Spark, specifically for similarity search with spatial data and IoT applicatons [75,76]. However, to date there is no clear implementation of more general MAM in modern distributed data processing frameworks.…”

Section: Scalable Similarity Searchmentioning

confidence: 99%

Unconventional application of k-means for distributed approximate similarity search

Vadillo-Ortega¹,

Algar²,

Diego³

et al. 2022

Preprint

View full text Add to dashboard Cite

Similarity search based on a distance function in metric spaces is a fundamental problem for many applications. Queries for similar objects lead to the well-known machine learning task of nearest-neighbours identification. Many data indexing strategies, collectively known as Metric Access Methods (MAM), have been proposed to speed up queries for similar elements in this context. Moreover, since exact approaches to solve similarity queries can be complex and time-consuming, alternative options have appeared to reduce query execution time, such as returning approximate results or resorting to distributed computing platforms. In this paper, we introduce MASK (Multilevel Approximate Similarity search with k-means), an unconventional application of the k-means algorithm as the foundation of a multilevel index structure for approximate similarity search, suitable for metric spaces. We show that inherent properties of k-means, like representing high-density data areas with fewer prototypes, can be leveraged for this purpose. An implementation of this new indexing method is evaluated, using a synthetic dataset and a real-world dataset in a high-dimensional and high-sparsity space. Results are promising and underpin the applicability of this novel indexing method in multiple domains.

show abstract

An efficient parallel indexing structure for multi-dimensional big data using spark

Cited by 7 publications

References 35 publications

A distributed computing model for big data anonymization in the networks

A distributed computing model for big data anonymization in the networks

Research on an evaluation index system of critical emergency management capability based on machine learning in a complex scientific environment

Unconventional application of k-means for distributed approximate similarity search

Contact Info

Product

Resources

About