Unsupervised Machine Learning based Documents Clustering in Urdu

Abstract

The volume of data on the web is growing rapidly, due to the proliferation of news sources, contents, blogs and journals etc. Like other languages, the Urdu language has also observed tremendous growth on the internet. As the volume of data is expanding, information retrieval (IR) is becoming complicated. Document clustering is an unsupervised ML approach, employed to group a huge number of dispersed documents into a small number of significant and consistent clusters, thus providing a base for indexing, IR and browsing mechanisms. Documents clustering has a long tradition in English as well as English like western languages, but Urdu lags behind in terms sophisticated natural language processing (NLP) tools and resources for documents clustering. Documents clustering becomes a challenging task in Urdu language having a rich morphology, particular structure, syntax peculiarities and cursive nature. In this study, we have developed a framework of document clustering and analysed various similarity measures for Urdu documents. We have also checked the effect of stop words removal in the process of Urdu document clustering.

Similarity Analysis in Understanding Online News in Response to Public Health Crisis

“…Recent studies have demonstrated effectiveness in clustering large volumes of online news items [ 14 , 15 , 16 ]. Text clustering can be defined as keeping similar group documents together, marking the outliers texts outside the group based on similarity estimation between them [ 17 ].…”

Building Dictionaries for Low Resource Languages: Challenges of Unsupervised Learning

“…The most popular part-of-speech tagging would be identifying words as nouns, verbs, adjectives, etc. The Albanian language has some properties that pose difficulties in creating a part-of-speech tag set [6]. A challenge faced in building a dictionary for low resource language (in this case for Albanian language) is that a partof-speech tag set that can adequately represent the underlying linguistic phenomena is difficult to build.…”