SmallClient for big data: an indexing framework towards fast data retrieval

Siddiqa, Aisha; Karim, Ahmad; Chang, Victor

doi:10.1007/s10586-016-0712-4

Cited by 16 publications

(9 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In terms of data indexing, SmallClient [11] improves query execution and search performance for big datasets and minimises the overhead of indexing. The framework is implementable on any distributed file system.…”

Section: B Data Indexing Techniquesmentioning

confidence: 99%

Integration of IoT Streaming Data With Efficient Indexing and Storage Optimization

et al. 2020

View full text Add to dashboard Cite

In the era of IoT, the world of connected experiences is created by the convergence of multiple technologies including real-time analytics, machine learning, and commodity sensors and embedded systems. However, with the proliferation of these IoT technologies and devices, there are challenges in integrating, indexing and managing time-series data from multiple sources to optimize the storage of those data and/or retrieve the information from them in real-time. Many researchers have addressed the data integration issue through developing time-series data compression techniques; however, they focused mainly on the application of integer value compression to IoT data. Moreover, existing work does not focus on the issues of data and information retrieval without decompression. In this paper, we solve these issues by constructing an indexing framework within a lossless compression for floating point time-series data, where an index is based on the time-stamp from the compressed data that facilitates the search for data without full decompression. We conduct several sets of experiments and quantify the performance of our proposed approach. The experimental results, performed on IoT datasets, show a reduction in storage compared with existing compression techniques. The experimental study also demonstrates the capability of time-series data indexing and integration in real-time. INDEX TERMS Data integration, indexing, time-series data compression, floating point compression, decompression, IoT streaming data, window-based compression and integration.

show abstract

Section: B Data Indexing Techniquesmentioning

confidence: 99%

Integration of IoT Streaming Data With Efficient Indexing and Storage Optimization

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Thus, the system overhead has occurred with massive data. Siddiqaet, A et al [5] designed a model named SmallClient for speeding the query for executing massive datasets. This method failed to use a probabilistic machine learning algorithm with B-Tree indexing for attaining adaptive index creation by predicting query workload and index attribute.…”

Section: Literature Reviewmentioning

confidence: 99%

“…•Indexing methods are effective for current datasets, but inefficient with big data. The indexing size and indexing time are essential for massive datasets, and thus it is unfeasible to tolerate long delays in uploading data and data search operations [5].…”

Section: Challengesmentioning

confidence: 99%

Optimization Driven MapReduce Framework for Indexing and Retrieval of Big Data

2020

KSII TIIS

View full text Add to dashboard Cite

With the technical advances, the amount of big data is increasing day-by-day such that the traditional software tools face a burden in handling them. Additionally, the presence of the imbalance data in big data is a massive concern to the research industry. In order to assure the effective management of big data and to deal with the imbalanced data, this paper proposes a new indexing algorithm for retrieving big data in the MapReduce framework. In mappers, the data clustering is done based on the Sparse Fuzzy-c-means (Sparse FCM) algorithm. The reducer combines the clusters generated by the mapper and again performs data clustering with the Sparse FCM algorithm. The two-level query matching is performed for determining the requested data. The first level query matching is performed for determining the cluster, and the second level query matching is done for accessing the requested data. The ranking of data is performed using the proposed Monarch chaotic whale optimization algorithm (M-CWOA), which is designed by combining Monarch butterfly optimization (MBO) [22] and chaotic whale optimization algorithm (CWOA) [21]. Here, the Parametric Enabled-Similarity Measure (PESM) is adapted for matching the similarities between two datasets. The proposed M-CWOA outperformed other methods with maximal precision of 0.9237, recall of 0.9371, F1-score of 0.9223, respectively.

show abstract

“…Researchers have used different indexing techniques on big data. For example, to improve query execution and the performance of searching [4], big data coupled with elastic search technology to satisfy the daily health care needs [5], a model with block creation module, index creation module and query creation [6], R-tree based indexing to support cloud with multi dimension data indexing, Data Med for searching biomedical datasets across repositories [7]. Similar to indexing,optimization is also one of searching techniques in big data for finding the best optimum.…”

Section: Introductionmentioning

confidence: 99%

Indexing and Optimization Techniques in Biomedical Industry

N.R*¹,

Prem²

2019

IJEAT

View full text Add to dashboard Cite

The unpredictable amount of data generated everyday by smart phones, social networks, health care systems etc. is really mind blowing. Smart phones alone generate 335exabytes of data ineveryyear that is really big data.Thus, the storage industry is facing several challenges in providing high magnitude of storage and retrieval devices at lowest costs which help to fulfill the requirements of big data and even technologies like de-duplication on storage devices are also becoming very important. Similarly, in recent days storing and retrieving the health care information in biomedical area is also becoming a great challenge in providing the best optimum data because of its huge amount of biomedical datasets. In order to achieve efficiency in providing highest quality health care information, an optimized index scheme is needed for big data which is based on accuracy and timelines. The existing indexing and optimization solutions are not enough to meet the emerging grow of index size and seek time. The objective of this paper is to identify better indexing solutions by investigating the basic big data requirements on indexing and optimization. This also includes a comparative study of various indexing and optimization techniques along with a taxonomy which contains Artificial Intelligence (AI) and Non Artificial Intelligence (NAI) based indexing techniques, optimization enhancement techniques which improves the performance efficiency of big data health care informatics.

show abstract

SmallClient for big data: an indexing framework towards fast data retrieval

Cited by 16 publications

References 27 publications

Integration of IoT Streaming Data With Efficient Indexing and Storage Optimization

Integration of IoT Streaming Data With Efficient Indexing and Storage Optimization

Optimization Driven MapReduce Framework for Indexing and Retrieval of Big Data

Indexing and Optimization Techniques in Biomedical Industry

Contact Info

Product

Resources

About