How Do Users Revise Zero-Hit Product Search Queries?

Amemiya, Yuki; Manabe, Takeshi; Fujita, Sumio; Sakai, Tetsuya

doi:10.1007/978-3-030-72240-1_14

Cited by 3 publications

(4 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The gen./specialization and aspect change transformations fall into the former type, whereas all other categories fall into the latter. We highlight here that unlike previous categorizations that describe how users revise queries in e-commerce [3,26], how to generate better queries to substitute the original query [28], how users reformulate queries in a session [27], we study here how to categorize query variations for the same information need which is a related but different problem.…”

Section: Uqv Taxonomymentioning

confidence: 99%

“…The three methods in this category add one spelling error to the query; the query term an error is introduced in is chosen uniformly at random. NeighbCharSwap Swaps two neighbouring characters from a random query term (excluding stopwords 3 ). RandomCharSub Replaces a random character from a random query term (excluding stopwords) with a randomly chosen new ASCII character.…”

Section: 21mentioning

confidence: 99%

See 1 more Smart Citation

Evaluating the Robustness of Retrieval Pipelines with Query Variation Generators

Penha¹,

Câmara²,

Hauff³

2021

Preprint

View full text Add to dashboard Cite

Heavily pre-trained transformers for language modelling, such as BERT, have shown to be remarkably effective for Information Retrieval (IR) tasks. IR benchmarks evaluate the effectiveness of (neural) ranking models based on the premise that a single query is used to instantiate the underlying information need. However, previous research has shown that (I) queries generated by users for a fixed information need are extremely variable and, in particular, (II) neural models are brittle and often easily make mistakes when tested with adversarial examples, i.e. examples with minimal modifications that do not change its label. Motivated by those observations we aim to answer the following question with our work: how robust are retrieval pipelines with respect to different variations in queries that do not change the queries' semantics? In order to obtain queries that are representative of users' querying variability, we first created a taxonomy based on the manual annotation of transformations occurring in a dataset (specifically UQV100) of user created query variations. For example, from the query 'cures for a bald spot' to the variation 'cures for baldness' we are applying a paraphrasing transformation that replaces words with synonyms. For each syntax-changing category of our taxonomy, we employ different automatic methods that when applied to a query generate a query variation. We conduct experiments on two datasets (TREC-DL-2019 and ANTIQUE) and create a total of 2430 query variations from 243 topics across both datasets. Our experimental results for two different IR tasks reveal that retrieval pipelines are not robust to query variations that maintain the content the same, with effectiveness drops of ∼20% on average when compared with the original query as provided in the datasets. Our findings indicate that further work is required to make retrieval pipelines with neural ranking models more robust and that IR collections should include query variations, e.g. using the methods proposed here, for a single information need to better understand models capabilities. The code and datasets are available at https://github.com/Guzpenha/query_variation_generators.

show abstract

Section: Uqv Taxonomymentioning

confidence: 99%

Section: 21mentioning

confidence: 99%

Evaluating the Robustness of Retrieval Pipelines with Query Variation Generators

Penha¹,

Câmara²,

Hauff³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The inter-annotator agreement [14] was moderate (Cohen's κ = 0.42); the disagreements were highest for the naturality and paraphrasing categories. We found that a total of 56 {q i , q j } pairs had more than one category assigned to it 3 . The resulting distribution is shown in Table 2 (right-most column); the categories of query variations that change the query without changing its semantics account for 57% of all the transformations.…”

Section: Paraphrasingmentioning

confidence: 88%

“…The gen./specialization and aspect change transformations fall into the former type, whereas all other categories fall into the latter. We highlight here that unlike previous categorizations that describe how users revise queries in e-commerce [3,24], how to generate better queries to substitute the original query [26], how users reformulate queries in a session [25], we study here how to categorize query variations for the same information need which is a related but different problem.…”

Section: Uqv Taxonomymentioning

confidence: 99%

Evaluating the Robustness of Retrieval Pipelines with Query Variation Generators

Penha

Câmara

Hauff

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Heavily pre-trained transformers for language modeling, such as BERT, have shown to be remarkably effective for Information Retrieval (IR) tasks, typically applied to re-rank the results of a first-stage retrieval model. IR benchmarks evaluate the effectiveness of retrieval pipelines based on the premise that a single query is used to instantiate the underlying information need. However, previous research has shown that (I) queries generated by users for a fixed information need are extremely variable and, in particular, (II) neural models are brittle and often make mistakes when tested with modified inputs. Motivated by those observations we aim to answer the following question: how robust are retrieval pipelines with respect to different variations in queries that do not change the queries' semantics? In order to obtain queries that are representative of users' querying variability, we first created a taxonomy based on the manual annotation of transformations occurring in a dataset (UQV100) of user-created query variations. For each syntax-changing category of our taxonomy, we employed different automatic methods that when applied to a query generate a query variation. Our experimental results across two datasets for two IR tasks reveal that retrieval pipelines are not robust to these query variations, with effectiveness drops of ≈ 20% on average. The code and datasets are available at https://github.com/Guzpenha/query variation generators.

show abstract