Site-Restricted Web Searches for Data Collection in Regional Dialectology

Abstract

This paper presents a new method for data collection in regional dialectology based on site-restricted web searches. The method allows for the values of many lexical alternation variables to be measured across a region of interest using common search engines such as Google or Bing. The method involves estimating the proportions of the variants of a lexical alternation variable over a series of cities by counting the number of webpages that contain these variants on newspaper websites originating from these cities through site-restricted web searches. The method is evaluated by mapping the 26 variants of 10 content word alternation variables with known distributions in American English. In almost all cases, the maps based on site-restricted web searches align closely with traditional dialect maps based on data gathered through questionnaires, demonstrating the accuracy of this method for the observation of regional linguistic variation. However, unlike collecting dialect data using traditional methods, which is a relatively slow process, the use of site-restricted web searches allows for dialect data to be collected from across a region as large as the United States in a matter of days.

Mapping Lexical Dialect Variation in British English Using Twitter

“…There is, however, no standard method for bivariate map comparison in dialectology. Other than visually comparing dialect maps (e.g., Grieve et al, 2013 ), the simplest approach is to correlate the two maps by calculating a correlation coefficient (e.g., Pearson's r ), essentially comparing the values of the two maps at every pair of locations. This was the approach taken in Grieve ( 2013 ), for example, where Pearson correlation coefficients were calculated to compare a small number of maps representing general regional patterns of grammatical and phonetic variation.…”

Geographical patterns of formality variation in written Standard California English

“…Despite the potential noise in the collected data, the site-restricted Web searches method was proved valid through an evaluation across the USA for lexical word alternation variables distribution attested by both this method and previous American English research. Site-restricted Web searches returned linguistic distribution results that were comparable to results obtained through traditional linguistic data collections (see details in Grieve et al, 2013). The validity of the method was therefore widely proven despite of the potential noise.…”

“…The most efficient technique up to date to gather linguistic frequencies from online texts is siterestricted Web searches (Grieve et al, 2013). Starting from a list of suitable newspaper Web sites based in the geographical area to be investigated and from a list of lexical alternation variables formed by variants denoting different degrees of formality, the Google search engine was queried for the number of hits for each variant of the selected variables in the entire archive of each newspaper.…”

Advances in Dialectometry

“…Grieve et al (2011) then used this data set to investigate American regional lexical variation. Grieve et al (2014) also focused on data from the web and introduced site-restricted web searches to dialectometry. Their approach consisted of searching (e.g., via Google) the websites of local newspapers in the United States for the occurrences of certain lexical alternations (e.g., 'bag' and 'sack').…”