Prediction of Agricultural Commodity Prices using Big Data Framework

Abstract

The agriculture sector plays a crucial role in the economy of Pakistan, contributing significantly to the Gross Domestic Product (GDP) and the employment rate. However, this sector faces challenges such as climate change, water scarcity, and low productivity, which have a direct impact on agricultural commodity prices. Accurate forecasting of commodity prices is essential for farmers, traders, and policymakers to make informed decisions and improve economic outcomes. This paper explores the use of a big data framework for agricultural commodity price forecasting in Pakistan, using a historical dataset on commodity prices in various Pakistani cities from 2007 to 2022 and Apache Spark to preprocess and clean the data. Based on historical spinach prices in Vehari City, the machine learning models Auto-Regressive Moving Average (ARIMA), Random Forest, and Long-Short-Term Memory (LSTM) were applied to price trends, and their performance was compared using Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE) and squared correlation coefficient (R2). LSTM outperformed ARIMA and Random Forest with a higher R2 value of 0.8 and the lowest MAE of 125.29. Such predictions can help farmers to effectively plan crop cultivation and traders to make well-informed decisions.