Stemming and n-grams in Spanish: an evaluation of their impact on information retrieval

Abstract

Abstract:At some stage, most of the models and techniques implemented in IR use frequency counts of the terms appearing in documents and in queries.However, many words, since they are derived from the same stem, have very close semantic contents. This makes a grouping of such variants under a single term advisable. Otherwise, dispersal occurs in the calculation of frequency of these terms, and it also becomes difficult to compare queries and documents. On the other hand, there are notable differences between different languages in the way of forming derivatives and inflected forms, so that the application of specific techniques can produce unequal results according to the language of the documents and queries. A description is given of the tests carried out for documents in Spanish, which involved some stemming techniques widely used in English, as well as the application of n-grams, and the results are compared.

On the Usefulness of Extracting Syntactic Dependencies for Text Indexing

“…It is difficult to know if this improvement is due to a more accurate extraction of pairs or due to differences between Spanish and English constructions. An important characteristic of the CLEF collection that can have a considerable impact on the performance of linguistically motivated indexing techniques is the large number of typographical errors present in documents, as have been reported in [8]. In particular, titles of the news (documents) are in capital letters without accents.…”

Reagrupamiento en familias y lexematización automática independientes del idioma

“…En [21] y [22] el lector puede hallar un estado del arte muy completo al respecto. Ejemplos de este tipo de análisis son la comparación de grafos [20], la utilización de n-gramas [16,34], la búsqueda de analogías [32], los modelos superficiales a base de reglas [38,31], los modelos probabilísticos [12], la segmentación por optimización [11,19], el aprendizaje no supervisado de las familias morfológicas por clasificación jerárquica ascendente [7], la lematización usando distancias de Levenshtein [14] o la identificación de sufijos por medio de la entropía [42]. Estos métodos se distinguen por el tipo de resultados obtenidos, ya sea la identificación de lemas, stems o sufijos.…”

Discovering Knowledge in a Large Organization through Support Vector Machines

“…We have also built a Lemmatizer to apply to documents from the collection object of our study, based on the works of Porter and other more specific to the Spanish language [10].…”