Large-scale multimedia content analysis using scientific workflows

Abstract

Analyzing web content, particularly multimedia content, for security applications is of great interest. However, it often requires deep expertise in data analytics that is not always accessible to non-experts. Our approach is to use scientific workflows that capture expert-level methods to examine web content. We use workflows to analyze the image and text components of multimedia web posts separately, as well as by a multimodal fusion of both image and text data. In particular, we re-purpose workflow fragments to do the multimedia analysis and create additional components for the fusion of the image and text modalities. In this paper, we present preliminary work which focuses on a Human Trafficking Detection task to help deter human trafficking of minors by thus fusing image and text content from the web. We also examine how workflow fragments save time and effort in multimedia content analysis while bringing together multiple areas of machine learning and computer vision. We further export these workflow fragments using linked data as web objects.

Cracking Sex Trafficking: Data Analysis, Pattern Recognition, and Path Prediction

“…The process, which is comprised of data collection, image segmentation or text processing, feature generation, clustering and classification, statistical feature selection, and fusion (Sethi et al. 2013, p. 817, figure 4), classified approximately 55% of the ads in a test sample correctly.…”

“…The bot crawls city-specific classified ad listings every 2 hours to identify trends. In a related study, Sethi et al (2013) describe a workflow system that aids in the fight against sex trafficking by analyzing multi-media content in an effort to identify ads involving minors. It incorporates the "trafficbot" approach for collecting classified ad data within the "wings" workflow system.…”

Automatic Understanding of Image and Video Advertisements

“…[82,48] determine the best placement of a commercial in a video stream, or of image ads in a part of an image using user affect and saliency. [64,19] detect whether the current video shown on TV is a commercial or not, and [65] detect human trafficking advertisements. [85] extract the object being advertised from commercials (videos), by looking for recurring patterns (e.g.…”

GPU-Enabled Serverless Workflows for Efficient Multimedia Processing

“…The adoption of cloud computing for multimedia processing is certainly another active field of study. Indeed, the paper by Sethi et al [13] already addressed the application of scientific workflows for multimedia content processing, leveraging the Wings [14] framework to efficiently analyse large-scale multimedia content across the Pegasus engine [15], which operates over cloud infrastructures. Another example of the adoption of cloud technologies in this area is the study conducted by Xu et al [16], where they propose the development of a workflow scheduling system for cloudlets based on Blockchain, ensuring the QoS of multimedia applications in these small-scale data centres near the edge.…”