What you see depends on where you sit: The effect of geographical location on web‐searching for systematic reviews: A case study

Abstract

There is limited guidance on how to web‐search in systematic reviews and concern relates to the reproducibility of searches using search engines such as Google. The aim of this paper is to address one potential source of variation in Google searches: does the geographical location of a researcher affect Google search returns? Using a virtual private network, we ran the same web‐search for the medical technology Dasatinib in 12 different countries. Two researchers independently extracted the search returns by country organised by page rank. We compared: C1. any difference in the items returned by Google searches between countries and C2. any difference in the page rank of items returned between countries. Searches were undertaken on Monday September 28th 2020. From 12 countries, 43 items were identified. For C1: 19 items were common to all 12 countries. Twenty‐four items were missed by searches in some countries. This means that there were differences in search returns between countries. For C2: a randomised trial reported by Raddich et al was the first search return for all countries. All other items, common to all countries, varied in their page‐rank. We find that geographic location would appear to influence Google search returns based on the findings of this case study. The findings suggest that recording the location of the researcher undertaking web‐searching may now be an important factor to report alongside detail on steps taken to minimise personalisation of web‐searches covered by recent guidance. This finding also has implications for stopping‐rules.

Irreproducibility in searches of scientific literature: A comparative analysis

“…Indeed, even the “Core Collection” of Web of Science, which we queried in our study, consists of several databases that may fall under different licenses in institutional subscriptions (Gusenbauer & Haddaway, 2019; Liu, 2019). On the other hand, Google Scholar is likely to be similar to the main Google search in its geographical dependencies when providing search result (Cooper et al., 2021). Overall, in our tests, Scopus and PubMed proved to be the most consistent databases, and Web of Science and Google Scholar produced less consistent results.…”

“…Yet, knowledge of the consistency within each search platform in relation to the location (i.e. institutional server) where the search is requested from, software environment, or computer configuration remain surprisingly limited (but see Gusenbauer & Haddaway, 2019 for location consistencies of scientific search platforms and Cooper et al., 2021 for geographic varions in Google search results). Since the search histories of users may be stored in the browsers’ cache and considered by the scientific search platforms, repeated and identical searches may result in different outcomes.…”

Exploring the relevance of the effect of geographical location when searching for studies using Google Search

“…Exploring the relevance of the effect of geographical location when searching for studies using Google Search I was interested to read the case study by Cooper et al on the effect of geographical location when using Google Search to identify studies for a systematic review. 1 The findings presented on the extent of difference in the search results retrieved in different geographical locations improves our understanding of the challenges of using Google Search for a systematic review. In many ways this is an excellent study and I am grateful to the authors for investigating this issue.…”

Response to: Briscoe “Exploring the relevance of the effect of geographical location when searching for studies using Google Search”

“…We thank Mr. Briscoe for his interest in our research note. 1 The evaluation of effectiveness that Mr. Briscoe sets out was not a part of the research aim in our research note. Our aim was to explore a question of general applicability, on which there is limited empirical data, namely: Does the geographical location of a researcher affect Google search returns when undertaking a websearch for a medical technology?…”

“…Whilst the aim of our research note was to question if geographical location might affect Google searches, our wider interest was to see how any 'effect' might be used. 1 We see a place for websearching as a secondary search method in complex reviews. We have already described and illustrated a conceptual process model which we seek to test in future work (see Figure 1).…”