Deriving query intents from web search engine queries

Abstract

The purpose of this paper is to test the reliability of query intents derived from queries, either by the user who entered the query or by another juror. We report the findings of three studies:First, we conducted a large-scale classification study (approximately 50,000 queries) using a crowdsourcing approach. Then, we used click-through data from a search engine log and validated the judgments given by the jurors from the crowdsourcing study. Finally, we conducted an online survey on a commercial search engine's portal. Since we used the same queries for all three studies, we were able to compare the results and the effectiveness of the different approaches, as well. We found that neither the crowdsourcing approach using jurors who classified queries originating from other users, nor the questionnaire approach using searchers who were asked about their own query that they just entered into a web search engine, lead to satisfying results. This leads us to conclude that there is little understanding of the classification tasks, even though both groups of jurors were given detailed instructions.While we used manual classification, our research has important implications for automatic classification, as well. We must question the success of approaches using automatic classification and comparing its performance to a baseline from human jurors. Running head: DERIVING QUERY INTENTS 2Keywords: search engines, information needs, query classification, user intent, web queries, web searching Deriving Query Intents from Web Search Engine QueriesSearch engines are by far the major means to finding information on the Web. In just one month, 131 billion queries were posed to the general-purpose search engines (ComScore, 2010). Studies report that the performance of search engines in terms of retrieval effectiveness is only moderate (Bar-Ilan, Keenoy, Yaari, & Levene, 2007;Griesbaum, 2004;Véronis, 2006;Lewandowski, 2008), and search engines' responses to more complex search tasks like exploratory searches (Marchionini, 2006) are considered poor (Singer, Norbisrath, Lewandowski, Vainikko, & Kikkas, 2011). However, when considering simpler tasks such as finding home pages, the performance of search engines is quite good (Lewandowski, 2011).It is clear that first, the quality of search engines' results depends on the difficulty of the task, and second, that search engines perform better on simpler tasks. To identify task types and to provide the best suitable results for the individual tasks, it is important to identify users' intent behind their queries.In this paper, we focus on identifying the following query intents: 1.Informational, where the user aims at finding some documents on a topic he is interested in.2. Navigational, where the user aims at navigating to a web page already known or where the user at least assumes that a specific web page exists. 3.Transactional, where the user wants to find a web page where a further transaction (e.g., downloading software, playing a game) can be performed. 4.Commercial, wher...

Evaluating the retrieval effectiveness of web search engines using a representative query sample

“… Transactional queries, where the user wants to find a Web page where a further transaction (e.g., downloading software, playing a game) can take place. While there have been attempts to improve this classification (mainly through adding sub-categories), for most purposes, the basic version still allows for a sufficiently differentiated view (for a further discussion on query intents and their usage in search engines, see Lewandowski, Drechsler, & Mach (2012). The vast majority of search engine retrieval effectiveness studies focus on informational queries (see table 1, and next section).…”

Reliability and validity of query intent assessments

“…Instead, we chose to limit our subject pool to colleagues, all computer scientists. As a result, our data set is small compared with previously collected query labeling data sets (Baeza‐Yates et al., ; Calderón‐Benavides et al., ; Lewandowski et al., ). However, the value of our data collection is not in its size, but in the fact that all queries have been labeled according to their intent, by the searchers themselves as well as at least two external assessors.…”

“…Similar to Lewandowski et al. (), we measure the agreement between the external assessors and the searchers, but we use a classification scheme with multiple dimensions instead of the Broder dimension only, which allows us to draw more detailed conclusions on the potential of automatic intent classification.…”

“…In addition to the work by White et al (2010), we not only collect queries but also explicit intent annotations according to a multidimensional intent classification scheme. Similar to Lewandowski et al (2012), we measure the agreement between the external assessors and the searchers, but we use a classification scheme with multiple dimensions instead of the Broder dimension only, which allows us to draw more detailed conclusions on the potential of automatic intent classification.…”

“…Queries were displayed in chronological order. Similar to the work by Lewandowski et al (2012), participants were allowed to select more than one value in a dimension.…”