Evaluation of stop word lists in text retrieval using Latent Semantic Indexing

Zaman, A. N. K.; Matsakis, Pascal; Brown, Charles Grant

doi:10.1109/icdim.2011.6093315

Cited by 28 publications

(11 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Examples are articles (e.g., a, an, the), prepositions (e.g., at, by, in, to, from, with) and conjunctions (e.g., and, but, as, because) [13]. As mentioned by Zaman et al [14], removal of stop words plays an important role in different text processing domains and it can be used to get language relations [15]. This was attested by Shin et al [16], as they mentioned that stop word removal improves the quality of the indexed terms.…”

Section: Review Of Related Literaturesmentioning

confidence: 99%

Analysis on the use of Latent Semantic Indexing (LSI) for document classification and retrieval system of PNP files

Aquino

Chavez

2018

MATEC Web Conf.

View full text Add to dashboard Cite

Document classification is the process of categorizing documents from many mixed files automatically [1]. In this paper, an approach to classification of documents for admin-case files of Philippine National Police (PNP) using Latent Semantic Indexing (LSI) method is proposed. The model for this that represents term-to-term, document-todocument and term-to-document relationships has been applied. Regular Expression is implemented also to define a search pattern based on character strings which the LSI used to establish the semantic relevance of the character strings to the search term or keyword. The aim of the study is to evaluate the performance of LSI in classifying PNP documents; experimentation was done using software to test the capability of LSI towards text retrieval. Indexing is according to the pattern matched in the collection of text that uses model of SVD. Based on tests, documents were indexed based on file relationships and was able to return a search result as the retrieved information from PNP files. Weights are used to check the accuracy of the method; the positive values identified in query similarity are regarded as the most relevant among the related searches, meaning, the query word matches words in a text file and it returns a query result.

show abstract

Section: Review Of Related Literaturesmentioning

confidence: 99%

Analysis on the use of Latent Semantic Indexing (LSI) for document classification and retrieval system of PNP files

Aquino

Chavez

2018

MATEC Web Conf.

View full text Add to dashboard Cite

show abstract

“…Table (1) shows in detail the statistical distribution of words in the data sets ( L is the average length of the document and L the standard deviation of document length). We used the Glasgow list [8] as a stop words list in our experiments. This list is widely used as English standard stop word; it covers a large number (351 stop words).…”

Section: Pretreatmentmentioning

confidence: 99%

A Semantic-based Variables Selection for Ontology Learning Taking Jaccard Alignment as Case

Djellali¹

2014

Procedia Computer Science

View full text Add to dashboard Cite

In the past decade, research on numerical schemes on ontology learning has been quite intensive. Several learning approaches have been proposed to help developers during the maintenance process. Most of the proposed approaches do not process the curse of dimensionality and the semantic contained in the information structure. A novel semantic-based method for ontology learning, which can provide improvement in both alignment and learning, is described. Good comparisons with the experimental studies demonstrate the multidisciplinary applications of our approach.

show abstract

“…Filtering stopwords is an established technique in Information Retrieval (IR). In many domains stopwords reduce noise and increase the efficiency of an inverted index [6,21]. In this paper we examine the use of stopwords, namely filtering out terms from the scoring decision, in neural re-ranking models.…”

Section: Introductionmentioning

confidence: 99%

“…Term importance and stopwords have a long history in IR: from the collection occurrence based Inverse Document Frequency (IDF) [17], the theory of retrievability [1,13], to studies on the impact of stopwords in retrieval and text processing [6,21], and visual inspection tools of neural re-ranking models [10]. Term importance in neural models has been studied in the context of query attention [7,19] and IDF weighting [11], or BERT-based importance measures for indexing [4].…”

Section: Introductionmentioning

confidence: 99%

Learning to Re-Rank with Contextualized Stopwords

Hofstätter

Lipani

Zlabinger

et al. 2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

The use of stopwords has been thoroughly studied in traditional Information Retrieval systems, but remains unexplored in the context of neural models. Neural re-ranking models take the full text of both the query and document into account. Naturally, removing tokens that do not carry relevance information provides us with an opportunity to improve the effectiveness by reducing noise and lower document representation caching-storage requirements. In this work we propose a novel contextualized stopword detection mechanism for neural re-ranking models. This mechanism consists of training a sparse vector in order to filter out document tokens from the ranking decision. This vector is learned end-to-end based on the contextualized document representations, allowing the model to filter terms on a per occurrence basis. This leads to a more explainable model, as it reduces noise. We integrate our component into the state-of-the-art interaction-based TK neural re-ranking model. Our experiments on the MS MARCO passage collection and queries from the TREC 2019 Deep Learning Track show that filtering out traditional stopwords prior to the neural model reduces its effectiveness, while learning to filter out contextualized representations improves it.

show abstract

Evaluation of stop word lists in text retrieval using Latent Semantic Indexing

Cited by 28 publications

References 10 publications

Analysis on the use of Latent Semantic Indexing (LSI) for document classification and retrieval system of PNP files

Analysis on the use of Latent Semantic Indexing (LSI) for document classification and retrieval system of PNP files

A Semantic-based Variables Selection for Ontology Learning Taking Jaccard Alignment as Case

Learning to Re-Rank with Contextualized Stopwords

Contact Info

Product

Resources

About