Accurate, Fast, But Not Always Cheap: Evaluating “Crowdcoding” as an Alternative Approach to Analyze Social Media Data

Abstract

Crowdcoding, a method that outsources “coding” tasks to numerous people on the internet, has emerged as a popular approach for annotating texts and visuals. However, the performance of this approach for analyzing social media data in the context of journalism and mass communication research has not been systematically assessed. This study evaluated the validity and efficiency of crowdcoding based on the analysis of 4,000 tweets about the 2016 U.S. presidential election. The results show that compared with the traditional quantitative content analysis, crowdcoding yielded comparably valid results and was superior in efficiency, but was more expensive under most circumstances.

Computer-assisted digital text analysis for journalism and communications research: introducing corpus linguistic techniques that do not require programming

“…Above, we have aimed to demonstrate that corpus linguistic analysis can be useful for the analysis of preferred and dispreferred language, sources, stigma and responsibility, framing, and project-specific text analysis. It has become apparent that this approach can be used for analysis of ‘manifest content’ (surface content, for example, most frequent words) and of ‘latent content – the underlying meaning of a message’ (Guo et al, 2019: 5). In addition to English, a wide range of other languages is supported.…”

“…It is clear that different approaches to text analysis have different advantages and limitations. For example, quantitative content analysis has limitations in relation to large datasets, both in terms of efficiency and cost, while fully automated NLP methods require the development and training of computer models, and computer-generated results can have low accuracy (Guo et al, 2019). Outsourcing coding to trained Internet workers can be ethically problematic (Williamson, 2016) or expensive (Guo et al, 2019).…”

“…For example, quantitative content analysis has limitations in relation to large datasets, both in terms of efficiency and cost, while fully automated NLP methods require the development and training of computer models, and computer-generated results can have low accuracy (Guo et al, 2019). Outsourcing coding to trained Internet workers can be ethically problematic (Williamson, 2016) or expensive (Guo et al, 2019). The key advantages and limitations of corpus linguistic analysis (in the form of the fairly basic techniques introduced above) are summarized in Table 4.…”

“…While our case study made use of a dataset of news items, techniques such as identifying highly frequent words or examining how words are used that refer to particular concepts are arguably useful for a wider range of data, from government reports, to policy documents, to social media data. As mentioned, future research needs to be undertaken to directly compare corpus linguistic text analysis with other approaches such as purely manual text/content/framing analysis (undertaken ‘by hand’) or indeed other computer-assisted and computational approaches for its validity, reliability, efficiency, and cost (à la Guo et al, 2019 for crowdcoding). Practitioner reports from communications scholars who have learnt and used the approach would also be valuable for better understanding of the potential barriers to using corpus linguistics in this field.…”

“…Kananovich, 2018), sentiment analysis of Facebook posts (e.g. Ji et al, 2019), ‘crowdcoding’ for analysis of political sentiment on Twitter (Guo et al, 2019), or a myriad of other techniques (see Boumans and Trilling, 2016), that digital methods are upon us is undeniable. However, the barrier for undertaking such digital projects is arguably still relatively high.…”