Sliding into My DMs: Detecting Uncomfortable or Unsafe Sexual Risk Experiences within Instagram Direct Messages Grounded in the Perspective of Youth

Abstract

We collected Instagram data from 150 adolescents (ages 13-21) that included 15,547 private message conversations of which 326 conversations were flagged as sexually risky by participants. Based on this data, we leveraged a human-centered machine learning approach to create sexual risk detection classifiers for youth social media conversations. Our Convolutional Neural Network (CNN) and Random Forest models outperformed in identifying sexual risks at the conversation-level (AUC=0.88), and CNN outperformed at the message-level (AUC=0.85). We also trained classifiers to detect the severity risk level (i.e., safe, low, medium-high) of a given message with CNN outperforming other models (AUC=0.88). A feature analysis yielded deeper insights into patterns found within sexually safe versus unsafe conversations. We found that contextual features (e.g., age, gender, and relationship type) and Linguistic Inquiry and Word Count (LIWC) contributed the most for accurately detecting sexual conversations that made youth feel uncomfortable or unsafe. Our analysis provides insights into the important factors and contextual features that enhance automated detection of sexual risks within youths' private conversations. As such, we make valuable contributions to the computational risk detection and adolescent online safety literature through our human-centered approach of collecting and ground truth coding private social media conversations of youth for the purpose of risk classification.

Desperate or desirable? Perceptions of individuals seeking dates online and offline

“…In this context, individuals who use social media to initiate romantic contact are often sending personal messages to strangers or acquaintances who may not be receptive to the unsolicited contact. Private communications of this sort may become sexual harassment, leading the receiver to feel uncomfortable and unsafe (Razi et al, 2023;Salerno-Ferraro et al, 2022), which would, in turn, reflect poorly on the individual making the advance. In other words, users of online methods to find dates are generally viewed negatively, but those using social media to seek partners are seen as especially undesirable.…”

More Tools, More Control: Lessons from Young Users on Handling Unwanted Messages Online

“…Our research questions, listed below, centered on direct messaging apps and features. Much less research has focused solely on private interpersonal interactions online, despite the influential role they play and the possible risks they impose on young people in the U.S. (Razi et al, 2023):…”

Wrist-bound Guanxi, Jiazu, and Kuolie: Unpacking Chinese Adolescent Smartwatch-Mediated Socialization

“…In this paper, we chose XTC as our research context, aligning with a prevalent approach seen in many HCI interview-based papers that focus on popular platforms [107,123,161]. This research approach deepens our understanding of emerging and unexplored phenomena or domains.…”