The new profiling: Algorithms, black boxes, and the failure of anti-discriminatory safeguards in the European Union

Abstract

This article argues that with increasingly large databases and computational power, profiling as a key part of security governance is experiencing major changes. Targeting mobile populations in order to enact security via controlling and sifting the good from the bad, profiling techniques accumulate and process personal data. However, as advanced algorithmic analytics enable authorities to make sense of unprecedented amounts of information and derive patterns in a data-driven fashion, the procedures that bring risk into being increasingly differ from those of traditional profiling. While several scholars have dealt with the consequences of black-boxed and invisible algorithmic analytics in terms of privacy and data protection, this article engages the effects of knowledge-generating algorithms on anti-discriminatory safeguards. Using the European-level efforts for the establishment of a Passenger Name Record (PNR) system as an example, and on the theoretical level connecting distinct modes of profiling with Foucauldian thought on governing, the article finds that with pattern-based categorizations in data-driven profiling, safeguards such as the Charter of Fundamental Rights of the European Union or the EU data-protection framework essentially lose their applicability, leading to a diminishing role of the tools of the anti-discrimination framework.

From Individual to Group Privacy in Big Data Analytics

“…Shared ownership of identity is largely ignored in data protection law (cf. Leese 2014;Hildebrandt 2011;Floridi 2012;Taylor et al 2017). Existing legal protections reflect piecemeal responses to particular egregious uses of shared offline identifiers in decision-making, seen, for instance, in US anti-discrimination law (Barocas 2014) or the ban on personalised insurance premiums based on risk profiling in the EU (Newell and Marabelli 2015).…”

“…Personalisation segments a population so that only some groups are worthy of receiving certain opportunities or information, re-enforcing existing social (dis)advantages. Personalisation systems create self-fulfilling behaviours and limit the opportunities available to users according to their classification within the system (Macnish 2012;Leese 2014). Personalisation Binvolves unseen, categorical, computational judgments about which searches, articles, or purchases should probably come next^ (Ananny 2016, 103).…”

“…Systematic identification of the limitations of recorded data, or the set of observables that inform algorithmic classification, and how these limitations are reflected in models and decisions produced with analytics (Grindrod 2014) are hindered by the opacity of analytics (Burrell 2016). Ad hoc groups are volatile (Leese 2014); the subject's representation in a classification scheme evolves over time, with new labels applied, tweaked and removed as patterns are identified from new inputs. Memberships can similarly change quickly.…”

“…Members of ad hoc groups will struggle to become self-aware regarding the group's existence due to the opacity, secrecy and ubiquity of analytics in information societies (cf. Leese 2014;Hildebrandt 2008). It is for this reason that ad hoc grouping poses a different challenge to identity management than membership of offline collectives or ascriptive groups (see Section 4.1) in which self-awareness and collective agency are possible.…”

Risk Based Passenger Screening in Aviation Security: Implications and Variants of a New Paradigm

“…Furthermore, opaque decision and classification criteria may undermine the legal protection against discrimination (Leese 2014c). As has been discussed in the research literature, 'known traveler' programmes like 'Pre' may additionally reproduce socio-economic inequalities as they transfer differentiations and classifications from the private economy sector to the public security sector (Leese 2014b: 47).…”