Designing Toxic Content Classification for a Diversity of Perspectives

Abstract

In this work, we demonstrate how existing classifiers for identifying toxic comments online fail to generalize to the diverse concerns of Internet users. We survey 17,280 participants to understand how user expectations for what constitutes toxic content differ across demographics, beliefs, and personal experiences. We find that groups historically at-risk of harassment-such as people who identify as LGBTQ+ or young adults-are more likely to to flag a random comment drawn from Reddit, Twitter, or 4chan as toxic, as are people who have personally experienced harassment in the past. Based on our findings, we show how current one-size-fits-all toxicity classification algorithms, like the Perspective API from Jigsaw, can improve in accuracy by 86% on average through personalized model tuning. Ultimately, we highlight current pitfalls and new design directions that can improve the equity and efficacy of toxic content classifiers for all users.

Jury Learning: Integrating Dissenting Voices into Machine Learning Models

“…For the jury composition task, we displayed one of 5 possible comment sets (generated by random samples from our comment toxicity dataset [49] stratified by toxicity severity and labeler disagreement) to exemplify the type of content they would need to moderate on YourPlatform. Participants were then shown a simplified jury composition input form that allowed them to allocate 12-person jury slots using three demographic attributes: (1) gender (Female, Male, Non-binary, Other), ( 2) race (Black or African American, White, Asian, Hispanic, American Indian or Alaska Native, Native Hawaiian or Pacific Islander, Other) and (3) political affiliation (Conservative, Liberal, Independent, Other).…”

“…Moderators of online communities (N=18) were asked to author juries for a comment toxicity classification task. We find that the resulting juries contain 2.9 times the representation of non-White jurors and 31.5 times the representation of non-binary jurors compared to those created implicitly by a large toxicity dataset [49]. This increased diversity in the jury composition changed the algorithm's classifications on 14% of items, reflecting the fact that jury learning captured those individual jurors' views far better than a baseline, state of the art aggregated model (with an MAE of 0.62 versus 1.05).…”

“…Indeed, a toxicity model tuned with a simple positive or negative offset (i.e. baseline) for each annotator achieves far more accurate perannotator results than a standard classifier [49].…”

“…Jury learning enables the creation of many possible classifiers from a single dataset of labels, with the added requirement that the dataset contain information about each annotator for any group or voice that the practitioner wishes to include. For instance, the toxicity dataset we use as our example application domain [49] includes education, past work experience or qualifications, racial identity, gender identity, political affiliation, age, disability status. Practitioners specify jurors either individually, or using any group membership criteria available in the dataset.…”

“…4.1.1 Dataset description. We train our model using a publicly available balanced dataset [49] in which 107,620 social media comments were labeled by five annotators each, from a pool of 17,280 unique annotators. This dataset was collected to understand how user expectations for what constitutes toxic content differ across demographics, beliefs, and personal experiences.…”