Evaluation of Google question-answering quality

Abstract

Purpose The purpose of this paper is to evaluate Google question-answering (QA) quality. Design/methodology/approach Given the large variety and complexity of Google answer boxes in search result pages, existing evaluation criteria for both search engines and QA systems seemed unsuitable. This study developed an evaluation criteria system for the evaluation of Google QA quality by coding and analyzing search results of questions from a representative question set. The study then evaluated Google’s overall QA quality as well as QA quality across four target types and across six question types, using the newly developed criteria system. ANOVA and Tukey tests were used to compare QA quality among different target types and question types. Findings It was found that Google provided significantly higher-quality answers to person-related questions than to thing-related, event-related and organization-related questions. Google also provided significantly higher-quality answers to where- questions than to who-, what- and how-questions. The more specific a question is, the higher the QA quality would be. Research limitations/implications Suggestions for both search engine users and designers are presented to help enhance user experience and QA quality. Originality/value Particularly suitable for search engine QA quality analysis, the newly developed evaluation criteria system expanded and enriched assessment metrics of both search engines and QA systems.

Direct Answers in Google Search Results

“…It was found that Google provided significantly higher-quality answers to person-related questions than to thing-related, event-related, and organization-related questions. Google also provided significantly higher-quality answers to wherequestions than to who-, what-and how-questions [27]. Snippets are also influencing partisanship.…”

“…Featured snippets are displayed in three different forms, i.e. paragraphs [27], tables [23], or lists, either ordered or unordered [46]. In the beginning of featured snippets' introduction into SERPs, Miklosik et al [46] examined the different possible factors for presenting a piece of information from a particular webpage in featured snippets.…”

“…The observation of search engine algorithm history shows that the featured snippet form will be the most expected in the Google development process. In connection with the query, it gives a result similar to the conversation in natural language [27] and the snippet content is possible to be read at loud and it's suitable for voice search responses [47]. Researches on the way of entering a query to the search engine that causes the return of specific results in Google have led to the typology of queries.…”

Featured Snippets Results in Google Web Search: An Exploratory Study

“…Direct answer supposed to deliver answers for keywords, without need to visit the result presented in search engine [17]. This snippet is displayed in four different forms like paragraph [18], table [19] and ordered or unordered list [20].…”

The Snippets Taxonomy in Web Search Engines

“…Direct answer supposed to deliver answers for queries, without need to visit the result presented in search engine [13]. This snippet can be presented in several different forms like paragraph [14], table [15] and ordered or unordered list.…”