Evolving Lucene search queries for text classification

Hirsch, Laurence; Hirsch, Robin; Saeedi, Masoud

doi:10.1145/1276958.1277279

Cited by 13 publications

(5 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Luo (Luo and Zincir-Heywood 2006) describes a system where recurrent linear GP is used to classify documents that are encoded as word sequences. Genetic methods have also been used to induce rules or queries useful for classifying online text (Smith and Smith 1997;Hirsch et al 2007;Pietramala et al 2008). In this case, the evolution requires a fitness test based on some measure of classification accuracy.…”

Section: Genetic Methods In Text Classificationmentioning

confidence: 99%

A Comparison of Lucene Search Queries Evolved as Text Classifiers

Hirsch

Brunsdon

2018

Applied Artificial Intelligence

Self Cite

View full text Add to dashboard Cite

In this article, we use a genetic algorithm to evolve seven different types of Lucene search query with the objective of generating accurate and readable text classifiers. We compare the effectiveness of each of the different types of query using three commonly used text datasets. We vary the number of words available for classification and compare results for 4, 8 and 16 words per category. The generated queries can also be viewed as labels for the categories and there is a benefit to a human analyst in being able to read and tune the classifier. The evolved queries also provide an explanation of the classification process. We consider the consistency of the classifiers and compare their performance on categories of different complexities. Finally, various approaches to the analysis of the results are briefly explored.

show abstract

Section: Genetic Methods In Text Classificationmentioning

confidence: 99%

A Comparison of Lucene Search Queries Evolved as Text Classifiers

Hirsch

Brunsdon

2018

Applied Artificial Intelligence

Self Cite

View full text Add to dashboard Cite

show abstract

“…Para superar esta irregularidade, se faz uma categorização automática a partir do vocabulário matriz com técnicas de recuperação de informação. Apache Solr atua como motor de recuperação da informação: https://lucene.apache.org/solr Enquanto algumas propostas de classificação das ofertas de emprego recorrem à aprendizagem automática baseada no uso de máquinas vetoriais (Javed et al, 2015), preferimos aplicar algoritmos de classificação que constroem automaticamente consultas a serem executadas no Apache Solr (Hirsch et al, 2007;Sood et al, 2007;Cai et al, 2016). Se desenvolve um programa escrito em linguagem Python para pré-processar o texto dos anúncios, assim como para interagir com o Apache Solr durante a execução das consultas correspondentes e para elaborar com os resultados os conjuntos de dados.…”

Section: Processamento Da Informação Coletadaunclassified

Competências e habilidades digitais requeridas aos profissionais da informação nos anúncios brasileiros de emprego na web

Paletta¹,

Pastor-Sánchez

González

2021

EPI

View full text Add to dashboard Cite

We analyze online job ads to identify the digital competences that companies and organizations demand from information professionals. This information is obtained from a content analysis of online job ads. Their retrieval from the web is terminological, and specialists select ads that include knowledge, activities, or requirements related to information science and digital transformation. After the ad format is standardized, the ads are analyzed using an ad hoc taxonomy for categorization. The taxonomy and the corpus of the ads are compared using automatized XML files in Apache Solr. Based on the taxonomy, we approach the text of the ads. The obtained data are stored as CSV files, from which we generate the general and specific groups of knowledge. The results are displayed according to classes of knowledge and professional profiles, focusing on those that relate to digital transformation. We explain the activity branches and the transversal informatics knowledge of the companies offering the positions. The specific knowledge in the digital environment is then presented, interpreted, and grouped according to the ads’ most characteristic facets: digital objects, data banks; digital services; data analysis; knowledge banks and artificial intelligence; software; knowledge organization systems (KSO); rights and values; and web and portals. These facets are distinguished by their frequency and by the transformations they generate in professional activities. We conclude by considering the appearance of profiles that are quite removed from traditional denominations and activities, as well as considering the effects of digital transformation in a highly complex labor market and on the development of digital competences.

show abstract

“…Lucene uses a common data structure to accept the index input, so can be flexibly adapted to a variety of data sources, such as databases, office documents, PDF documents and html documents, etc., when data indexing, only needs an appropriate parser to convert the data source into the corresponding data structure. Although Lucene has powerful search and indexing capabilities, but it is not a complete search engine, cannot collect the information of Internet pages, and in sorting have yet to be perfected [8] . The sorting of search results is very important for the search engine, usually users only take attention to the first page search engine returned, therefore, taking the pages valuable for users, with high level as the top surface of the page is an important topic of search engine study.…”

Section: Technical Analysis Of Lucenementioning

confidence: 99%

Research and implementation of search engine based on Lucene

Wan¹,

Wang²

2016

Proceedings of the 2nd International Conference on Advances in Mechanical Engineering and Industrial Informatics (AMEII 2016)

View full text Add to dashboard Cite

Abstract. From in-depth research on the basic principles and architecture of search engine, through secondary development for Lucene development package, this paper designs an entire search engine system framework, and realizes its core modules. This system can make up for the deficiency of the existing Lucene framework, and enhance the accuracy of the search engine system, so has higher real and commercial value.

show abstract

Evolving Lucene search queries for text classification

Cited by 13 publications

References 19 publications

A Comparison of Lucene Search Queries Evolved as Text Classifiers

A Comparison of Lucene Search Queries Evolved as Text Classifiers

Competências e habilidades digitais requeridas aos profissionais da informação nos anúncios brasileiros de emprego na web

Research and implementation of search engine based on Lucene

Contact Info

Product

Resources

About