Der AMS-Algorithmus.

Bias does not equal bias: a socio-technical typology of bias in data-based algorithmic systems

“…This paper introduces the proposed bias typology along an Austrian case study of an algorithmic classification system for the unemployed, the AMS algorithm, that has been discussed widely due to its potential of discrimination (Allhutter et al, 2020;Kayser-Bril, 2019;Lopez, 2019;Szigetvari, 2018a;UN Special Rapporteur, 2019;Wagner et al, 2020;Wimmer, 2018b). Looking at this particular algorithmic system, it becomes especially clear that one has to strictly differentiate between two concepts I call socio-technical bias and societal bias.…”

“…The manifold criticism is centred around three focal points: firstly, it became known in a published method documentation of the algorithmic system that the personal data entry "Gender: Female" results in an automatic "deduction of points", which translates to the fact that unemployed individuals can be assigned to a less eligible group solely on the basis of their gender (Holl et al, 2018;Wimmer, 2018b). Further potential point deductions according to personal data entries, such as age, childcare responsibilities, disability and nationality, can lead to an intersectionally compounded effect (Lopez, 2019;Wagner et al, 2020): the group of job-seekers with the lowest "chances'' for job placement (according to the predictive model), group C, should not get full access to all AMS support resources, the rationale behind that being efficiency (Allhutter et al, 2020;Szigetvari, 2018a).…”

“…Accordingly, the typology presented in the following must always be considered together with a respective legal anti-dis- The respective legislator recognises that there are structural inequalities in society and deems them undesirable, thus prohibiting discriminatory treatment on the basis of this structural disadvantage (Givens & Evans Case, 2014). In Austria, for example, discrimination based on gender, parenthood, age, disability and various other so-called attributes in employment contexts is prohibited by law according to the GlBG which applies to the context of use of the AMS algorithm (Wagner et al, 2020).…”

“…In the case of the AMS algorithm, an example of socio-technical bias is the fact that the corresponding gender entry only knows two options, male and female, even though in Austria, the corresponding gender scheme is not binary (Allhutter et al, 2020;Wagner et al, 2020). The complexity of the, at least, legally possible gender variety, is not being captured by the possible categories in the AMS database.…”

“…With regard to the predictive health risk system, what is supposed to be measured, namely the actual need of a patient, is being approximated by the predicted cost in health care, revealing that Black patients have been and are being under-treated . With regard to the gender categories in the AMS system, the possible entries and therefore the degree to which complexity is enabled, are restricted to a binary gender scheme (Wagner et al, 2020).…”