Historical data based approach to mitigate stragglers from the Reduce phase of MapReduce in a heterogeneous Hadoop cluster

Bawankule, Kamalakant Laxman; Dewang, Rupesh Kumar; Singh, Anil Kumar

doi:10.1007/s10586-021-03530-x

Cited by 9 publications

(1 citation statement)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Certain studies have also opted for a web-based approach for visualizing and analyzing omics data [21][22][23][24]. In addition, due to COVID-19, there is active research and utilization of big data in the field of bioresearch [25][26][27][28][29]. With the recent availability of network-enabled devices that are equipped with web browsers as a standard feature, configuring the web as a user interface allows for consistent platform functionality across different devices.…”

Section: Related Workmentioning

confidence: 99%

Data Lake Conceptualized Web Platform for Food Research Data Collection

An,

Oh,

Kim

et al. 2024

JWE

View full text Add to dashboard Cite

Food research is uniquely intertwined with everyday life and necessitates the utilization of big data. Within this domain, the research data consist of various forms and formats, encompassing biological experiment results, chemical analysis data, nutritional information, microbiological data, sensor data, images, and videos. This diversity stems from the integration of data from various subdomains within the larger field. With recent advancements in deep learning technology, the importance of data has grown significantly, resulting in increased reliance on data-driven research. Although specialized platforms for sharing and utilizing data have been established at the national level, particularly in the bioscience field, food research lacks a dedicated infrastructure and specialized data-sharing platforms. In this study, we develop a platform that leverages Hadoop-based distributed file systems to create a data lake. This platform enables data storage and sharing through a web-based interface. The distributed file system supports scalability by adding data nodes, making it an effective solution for capacity expansion. In addition, the web-based platform ensures high accessibility, allowing users access from anywhere, at any time, using any device. Finally, we introduce the establishment of a 1.8 PB Hadoop-based physical storage system and present an approach for building a highly accessible web platform with substantial utility.

show abstract