CSMR: A Scalable Algorithm for Text Clustering with Cosine Similarity and MapReduce

Victor, Giannakouris-Salalidis; Plerou, Antonia; Sioutas, Spyros

doi:10.1007/978-3-662-44722-2_23

Cited by 11 publications

(6 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…After applying the SVD, the cosine similarity is calculated between user complaint and each agriculture problem to return the correct solution. The cosine [31] can be calculated as (4):…”

Section: Semantic Selection and Rankingmentioning

confidence: 99%

AgroSupportAnalytics: big data recommender system for agricultural farmer complaints in Egypt

Rslan

Khafagy²,

Ali³

et al. 2023

IJECE

View full text Add to dashboard Cite

<span>The world’s agricultural needs are growing with the pace of increase in its population. Agricultural farmers play a vital role in our society by helping us in fulfilling our basic food needs. So, we need to support farmers to keep up their great work, even in difficult times such as the coronavirus disease (COVID-19) outbreak, which causes hard regulations like lockdowns, curfews, and social distancing procedures. In this article, we propose the development of a recommender system that assists in giving advice, support, and solutions for the farmers’ agricultural related complaints (or queries). The proposed system is based on the latent semantic analysis (LSA) approach to find the key semantic features of words used in agricultural complaints and their solutions. Further, it proposes to use the support vector machine (SVM) algorithm with Hadoop to classify the large agriculture dataset over Map/Reduce framework. The results show that a semantic-based classification system and ﬁltering methods can improve the recommender system. Our proposed system outperformed the existing interest recommendation models with an accuracy of 87%.</span>

show abstract

“…After applying the SVD, the cosine similarity is calculated between user complaint and each agriculture problem to return the correct solution. The cosine [31] can be calculated as (4):…”

Section: Semantic Selection and Rankingmentioning

confidence: 99%

AgroSupportAnalytics: big data recommender system for agricultural farmer complaints in Egypt

Rslan

Khafagy²,

Ali³

et al. 2023

IJECE

View full text Add to dashboard Cite

show abstract

“…The number of times the feature f appears in query q is shown as tf (f,q) [17]. In this stage, the term frequency algorithm computes the weight of the word in the query vector using the frequency of each word [22]. Table 2 describes an example of the TF calculation for a simple query.…”

Section: Feature Weighting Based On Tf Using Mapreducementioning

confidence: 99%

“…where the weight of the words in each query vector is divided by the length of the words in that query. Algorithm 4 may be used to compute the Cosine similarity between the produced query vectors in parallel [22].…”

Section: Similarity Measurement Using Mapreducementioning

confidence: 99%

Performance Evaluation of Query Plan Recommendation with Apache Hadoop and Apache Spark

et al. 2022

View full text Add to dashboard Cite

Access plan recommendation is a query optimization approach that executes new queries using prior created query execution plans (QEPs). The query optimizer divides the query space into clusters in the mentioned method. However, traditional clustering algorithms take a significant amount of execution time for clustering such large datasets. The MapReduce distributed computing model provides efficient solutions for storing and processing vast quantities of data. Apache Spark and Apache Hadoop frameworks are used in the present investigation to cluster different sizes of query datasets in the MapReduce-based access plan recommendation method. The performance evaluation is performed based on execution time. The results of the experiments demonstrated the effectiveness of parallel query clustering in achieving high scalability. Furthermore, Apache Spark achieved better performance than Apache Hadoop, reaching an average speedup of 2x.

show abstract

“…Many work related to SVM in the parallel environment (or distributed system) have introduced in Ngoc et al [15], Wen et al [16], and Rao [17] . There are many researcher papers using Cloudera and Hadoop [18] Map/reduce. Studies using SVM [19] in parallel environment for semantic classification are proposed.…”

Section: Related Workmentioning

confidence: 99%

English Semantic Similarity based on Map Reduce Classification for Agricultural Complaints

Rslan¹,

Khafagy²,

Munir³

et al. 2021

IJACSA

View full text Add to dashboard Cite

Due to environmental changes, including global warming, climatic changes, ecological impact, and dangerous diseases like the Coronavirus epidemic. Since coronavirus is a hazardous disease that causes many deaths, government of Egypt undertook many strict regulations, including lockdowns and social distancing measures. These circumstances have affected agricultural experts' presence to help farmers or advise on solving agricultural problems. For helping this issue, this work focused on improving support for farmers on the major field crops in Egypt Retrieving solutions corresponding to farmer query. For our work, we have mainly focused on detecting the semantic similarity between large agriculture dataset and user queries using Latent Semantic Analysis (LSA) based on Term Frequency Weighting and Inverse Document Frequency (TF-IDF) method. In this research paper, we apply SVM MapReduce classifier as a framework for paralleling and distributing the work on the dataset to classify the dataset. Then we apply different approaches for computing the similarity of sentences. We presented a system based on semantic similarity methods and support vector machine algorithm to detect the similar complaints of the user query. Finally, we run different experiments to evaluate the performance and efficiency of the proposed system as the system performs approximately 77.8%~94.8% in F-score measure. The experimental results show that the accuracy of SVM classifier is approximately 88.68%~89.63% and noted the leverage of SVM classification to the semantic similarity measure between sentences.

show abstract

CSMR: A Scalable Algorithm for Text Clustering with Cosine Similarity and MapReduce

Cited by 11 publications

References 13 publications

AgroSupportAnalytics: big data recommender system for agricultural farmer complaints in Egypt

AgroSupportAnalytics: big data recommender system for agricultural farmer complaints in Egypt

Performance Evaluation of Query Plan Recommendation with Apache Hadoop and Apache Spark

English Semantic Similarity based on Map Reduce Classification for Agricultural Complaints

Contact Info

Product

Resources

About