Fields of Gold: Scraping Web Data for Marketing Insights

Abstract

Marketing scholars increasingly use web scraping and Application Programming Interfaces (APIs) to collect data from the internet. Yet, despite the widespread use of such web data, the idiosyncratic and sometimes insidious challenges in its collection have received limited attention. How can researchers ensure that the datasets generated via web scraping and APIs are valid? While existing resources emphasize technical details of extracting web data, the authors propose a novel methodological framework focused on enhancing its validity. In particular, the framework highlights how addressing validity concerns requires the joint consideration of idiosyncratic technical and legal/ethical questions along the three stages of collecting web data: selecting data sources, designing the data collection, and extracting the data. The authors further review more than 300 articles using web data published in the top five marketing journals and offer a typology of how web data has advanced marketing thought. The article concludes with directions for future research to identify promising web data sources and to embrace novel approaches for using web data to capture and describe evolving marketplace realities.

Service failures in times of crisis: An analysis of eWOM emotionality

“…The findings build on the extant literature in multiple ways. First, following the tradition of using large datasets consisting of Twitter data to analyze people’s emotions ( Boegershausen et al, 2022 ), a dataset of 327,205 tweets was utilized to examine the effects of the COVID-19 pandemic on consumers’ emotions. While a growing body of literature uses social media data to analyze phenomena related to COVID-19 (e.g., Abd-Alrazaq et al, 2020 , Meng et al, 2021 , Saura et al, 2022 , Wang et al, 2021 ), to the best of our knowledge, no such research has focused on consumer emotionality.…”

“…A total of 44.3 % of UK-based internet users were active on Twitter in January 2021 ( Kemp, 2021 ). Thus, in line with prior brand-related social media research (e.g., Borah et al, 2020 , Golmohammadi et al, 2021 , Lee, 2021 , Rust et al, 2021 ), Twitter was used as a data source for this study ( Boegershausen et al, 2022 ). Approximately 327,205 tweets were extracted that matched the search criteria (described in detail below) using the social listening tool Pulsar TRAC ( https://www.pulsarplatform.com ).…”

Propelling International Marketing Research with Geospatial Data

“…While Google Maps and Bing Maps do offer free application programming interface (API) access, they put limitations on the number of queries, which can significantly hamper any international research endeavor covering multiple countries, regions, or cities. Although scraping data from these services is technically possible, doing so violates their terms of service, presents a legal gray area, and is actively inhibited by the providers (Boegershausen et al 2022; Dalton 2015).…”

JM: Promoting Catalysis in Marketing Scholarship

“…Many of the marketing challenges we face present problems that require multiple disciplinary lenses. For example, studying the cultural underpinnings of authenticity (Thompson and Kumar 2022) or scraping web data with precision and integrity (Boegershausen et al 2022). For these and other complex problems, using only a marketing lens may result in incomplete and suboptimal solutions.…”