Meteorological Data Analysis Using MapReduce

Fang, Wei; Sheng, Victor S.; Wen, Xuezhi; Pan, Wubin

doi:10.1155/2014/646497

Cited by 15 publications

(8 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Wei Fang, V.S. Sheng, Xue Zhi Wen and Wubin Pan [14] designed and implemented SVM classification technique using Hadoop and MapReduce framework to predict the rainfall from large amount data. They utilized feature selection and reduction algorithm associated with the dataset.…”

Section: Related Workmentioning

confidence: 99%

An Adaptive Model for Forecasting Seasonal Rainfall Using Predictive Analytics

Reddy¹,

Sureshbabu²

2019

IJIES

View full text Add to dashboard Cite

India is a country which has exemplary climate circumstances comprising of different seasons and topographical conditions like high temperatures, cold atmosphere, and drought, heavy rainfall seasonal wise. These utmost varieties in climate make us exact weather prediction is a challenging task. Majority people of the country depend on agriculture. Farmers require climate information to decide the planting. Weather prediction turns into orientation in the farming sector to decide the start of the planting season and furthermore quality and amount of their harvesting. One of the variables are influencing agriculture is rainfall. The main goal of this project is early and proper rainfall forecasting, that helpful to people who live in regions which are inclined natural calamities such as floods and it helps agriculturists for decision making in their crop and water management using big data analytics which produces high in terms of profit and production for farmers. In this project, we proposed an advanced automated framework called Enhanced Multiple Linear Regression Model (EMLRM) with MapReduce algorithm and the Hadoop file system. In the proposed model (EMLRM) first, we stored the unstructured weather data in hadoop distributed file system (HDFS), process that stored data by using MapReduce Algorithm and build the rainfall prediction model by utilizing Multiple Linear Regression.We used climate data from IMD (Indian Metrological Department, Hyderabad) in 1901 to 2002 period. The experimental outcomes show that the EMLRM provided the lowest value of Root Mean Square Error (RMSE= 0.274) and Mean Absolute Error (MAE= 0.0745) compared with existing methods. The results of the analysis will help the farmers to adopt effective modeling approach for predicting long-term seasonal rainfall.

show abstract

Section: Related Workmentioning

confidence: 99%

An Adaptive Model for Forecasting Seasonal Rainfall Using Predictive Analytics

Reddy¹,

Sureshbabu²

2019

IJIES

View full text Add to dashboard Cite

show abstract

“…This is because there will be one entry per block which is 1000x3 (number of blocks x replication factor) and one entry per file which is 1000. Each block or file entry will take about 150bytes [6][7] [8].…”

Section: Small File Problemmentioning

confidence: 99%

solution for the future: small file management by optimizing Hadoop

Achandair¹,

Bourekkadi²,

Elmahouti³

et al. 2018

IJET

View full text Add to dashboard Cite

Hadoop Distributed File System (HDFS) is designed to reliably store very large files across machines in a large cluster. It is one of the most used distributed file systems and offer a high availability and scalability on low-cost hardware. All Hadoopframework have HDFS as their storage component. Coupled with map reduce, which is the processing component, HDFS and Map Reduce (a processing component) have become the standard platforms for any management of big data in these days. HDFS however, in terms of design has the ability to handle huge numbers of large files, but when it comes to its deployments to handle large amounts of small files it might not be very effective. This paper puts forward a new strategy of managing small files. The approach will consists of two principal phases. The first phase will deal with the consolidating of aaclients input files, storing it continuously in a particular allocated block, that is a SequenceFile format, and so on into the next blocks. In this way we avoid the use of multiple block allocations for different streams, this reduces calls for available blocks and also reduces the metadata memory on the NameNode. Note the reason for this is that groups of small files packaged in a SequenceFile on the same block require one entry instead of one of each small file. The second phase will involve analyzing the attributes of stored small files so they can be distributed them in a way that the most called files will be referenced by an additional index as a MapFile format to reduce the read throughput during random access.

show abstract

“…Fang et al [36] used large-scale meteorological datasets and an MK-means algorithm applied through MapReduce to manage meteorological information stored in high-cost servers over several years. Optimal K-centroids were calculated using the standard distance measure of similarity and the Euclidean distance.…”

Section: Related Workmentioning

confidence: 99%

Clustering large datasets using K-means modified inter and intra clustering (KM-I2C) in Hadoop

2017

View full text Add to dashboard Cite

Big data has become popular for processing, storing and managing massive volumes of data. The clustering of datasets has become a challenging issue in the field of big data analytics. The K-means algorithm is best suited for finding similarities between entities based on distance measures with small datasets. Existing clustering algorithms require scalable solutions to manage large datasets. This study presents two approaches to the clustering of large datasets using MapReduce. The first approach, K-Means Hadoop MapReduce (KM-HMR), focuses on the MapReduce implementation of standard K-means. The second approach enhances the quality of clusters to produce clusters with maximum intra-cluster and minimum inter-cluster distances for large datasets. The results of the proposed approaches show significant improvements in the efficiency of clustering in terms of execution times. Experiments conducted on standard K-means and proposed solutions show that the KM-I2C approach is both effective and efficient.

show abstract

Meteorological Data Analysis Using MapReduce

Cited by 15 publications

References 11 publications

An Adaptive Model for Forecasting Seasonal Rainfall Using Predictive Analytics

An Adaptive Model for Forecasting Seasonal Rainfall Using Predictive Analytics

solution for the future: small file management by optimizing Hadoop

Clustering large datasets using K-means modified inter and intra clustering (KM-I2C) in Hadoop

Contact Info

Product

Resources

About