Massively Scalable Image Processing on the HPCC Systems Big Data Platform

Hukkeri, Tanmay Sanjay; Shobha, G; Phal, Shubham Milind; Shetty, Jyothi; R, Yatish H; Mohammed, Naweed

doi:10.1145/3378936.3378978

Cited by 4 publications

(2 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We made this choice because such high-level proposals often lack details for meaningful comparison, and do not always reflect the feasibility of actual system implementation. For a more industrial point of view, we point the reader to papers like [59], [73], [125].…”

Section: Contributions and Outlinementioning

confidence: 99%

Data Lakes: A Survey of Functions and Systems

Hai

Koutras

Quix

et al. 2023

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

Data lakes are becoming increasingly prevalent for big data management and data analytics. In contrast to traditional 'schema-on-write' approaches such as data warehouses, data lakes are repositories storing raw data in its original formats and providing a common access interface. Despite the strong interest raised from both academia and industry, there is a large body of ambiguity regarding the definition, functions and available technologies for data lakes. A complete, coherent picture of data lake challenges and solutions is still missing. This survey reviews the development, architectures, and systems of data lakes. We provide a comprehensive overview of research questions for designing and building data lakes. We classify the existing approaches and systems based on their provided functions for data lakes, which makes this survey a useful technical reference for designing, implementing and deploying data lakes. We hope that the thorough comparison of existing solutions and the discussion of open research challenges in this survey will motivate the future development of data lake research and practice.

show abstract

Section: Contributions and Outlinementioning

confidence: 99%

Data Lakes: A Survey of Functions and Systems

Hai

Koutras

Quix

et al. 2023

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

show abstract

“…Its key advantages arise from its scalability, performance and usability. The platform serves as an alternative to existing big-data platforms such as apache hadoop, apache spark and data torrent RTS [3]. The platform is supported by its underlying programming language ECL, which is implicitly parallel and declarative in nature, and provides several constructs to simplify parallel compute operations.…”

Section: Introductionmentioning

confidence: 99%

Massively scalable density based clustering (DBSCAN) on the HPCC systems big data platform

Phal

Hukkeri

et al. 2021

IJ-AI

Self Cite

View full text Add to dashboard Cite

<span id="docs-internal-guid-919b015d-7fff-56da-f81d-8f032097bce2"><span>Dealing with large samples of unlabeled data is a key challenge in today’s world, especially in applications such as traffic pattern analysis and disaster management. DBSCAN, or density based spatial clustering of applications with noise, is a well-known density-based clustering algorithm. Its key strengths lie in its capability to detect outliers and handle arbitrarily shaped clusters. However, the algorithm, being fundamentally sequential in nature, proves expensive and time consuming when operated on extensively large data chunks. This paper thus presents a novel implementation of a parallel and distributed DBSCAN algorithm on the HPCC Systems platform. The algorithm seeks to fully parallelize the algorithm implementation by making use of HPCC Systems optimal distributed architecture and performing a tree-based union to merge local clusters. The proposed approach* was tested both on synthetic as well as standard datasets (MFCCs Data Set) and found to be completely accurate. Additionally, when compared against a single node setup, a significant decrease in computation time was observed with no impact to accuracy. The parallelized algorithm performed eight times better for higher number of data points and takes exponentially lesser time as the number of data points increases.</span></span>

show abstract

A generic parallel optimization framework for solving hard problems in optical networks

Bose³

et al. 2023

Computer Communications

View full text Add to dashboard Cite

Massively Scalable Image Processing on the HPCC Systems Big Data Platform

Cited by 4 publications

References 7 publications

Data Lakes: A Survey of Functions and Systems

Data Lakes: A Survey of Functions and Systems

Massively scalable density based clustering (DBSCAN) on the HPCC systems big data platform

A generic parallel optimization framework for solving hard problems in optical networks

Contact Info

Product

Resources

About