An agenda for addressing bias in conflict data

Abstract

With increased availability of disaggregated conflict event data for analysis, there are new and old concerns about bias. All data have biases, which we define as an inclination, prejudice, or directionality to information. In conflict data, there are often perceptions of damaging bias, and skepticism can emanate from several areas, including confidence in whether data collection procedures create systematic omissions, inflations, or misrepresentations. As curators and analysts of large, popular data projects, we are uniquely aware of biases that are present when collecting and using event data. We contend that it is necessary to advance an open and honest discussion about the responsibilities of all stakeholders in the data ecosystem – collectors, researchers, and those interpreting and applying findings – to thoughtfully and transparently reflect on those biases; use data in good faith; and acknowledge limitations. We therefore posit an agenda for data responsibility considering its collection and critical interpretation.

Political instability patterns are obscured by conflict dataset scope conditions, sources, and coding choices

“…A critically understudied issue in conflict data and analysis is the degree to which similar data collection mandates and practices are affected by the choices of the dataset creators. These choices of conceptual prioritization and representation, frequency of release, and information source catchments have drastic implications for whether global conflict trends are accurately represented (for example, see Miller et al, 2022). Observing each dataset's collection process-what we call 'conflict catchment' and 'source catchment' (see below)-we compare projects, allowing us to move forward with the establishment of standards of collection and transparency in conflict data use.…”

“…That missing source data reduces the validity of the collection. In short, while adapting a collection strategy to prioritize validity does alter the datagenerating process, not adapting the strategy, and prioritizing only internal reliability to a predetermined conflict catchment, also results in other biases; neither option is without bias (Miller et al, 2022). The Philippines, for example, is a case where conflict catchment priorities result in dramatically different assessments of violence patterns.…”

“…Source diversity is not solely about the scale and new media integration, but language diversity. The four datasets explored here produce information in English, and most prioritize English language international media, despite its known limitations (Miller et al, 2022;Demarest and Langer, 2022;Schrodt et al, 2001;Boschee et al, 2013). International, English language media report on different types of violence compared to local media and reporting, and frame conflicts with certain prejudices (Raleigh and Kishi, 2019).…”

“…In short, as countries are unique in both their conflict and media landscapes, and all sources contain some biases (Miller et al, 2022), the ideal constellation of sources captures how conflict varies across countries, and accounts for local information. Yet, the number of sources alone is not a metric of reliability, and no specific source type guarantees perfectly valid data for all events (Ortiz et al, 2005).…”

“…Changes to a conflict environment demand that sourcing strategies shift for information to remain valid and accurate, rather than solely reliable. While dynamic sourcing strategies may introduce certain biases because the datagenerating process changes within time-series data, avoiding the integration of newly available information introduces other significant biases (Miller et al, 2022).…”