The promise of zero-shot learning for alcohol image detection: comparison with a task-specific deep learning algorithm

Abstract

Exposure to alcohol content in media increases alcohol consumption and related harm. With exponential growth of media content, it is important to use algorithms to automatically detect and quantify alcohol exposure. Foundation models such as Contrastive Language-Image Pretraining (CLIP) can detect alcohol exposure through Zero-Shot Learning (ZSL) without any additional training. In this paper, we evaluated the ZSL performance of CLIP against a supervised algorithm called Alcoholic Beverage Identification Deep Learning Algorithm Version-2 (ABIDLA2), which is specifically trained to recognise alcoholic beverages in images, across three tasks. We found ZSL achieved similar performance compared to ABIDLA2 in two out of three tasks. However, ABIDLA2 outperformed ZSL in a fine-grained classification task in which determining subtle differences among alcoholic beverages (including containers) are essential. We also found that phrase engineering is essential for improving the performance of ZSL. To conclude, like ABIDLA2, ZSL with little phrase engineering can achieve promising performance in identifying alcohol exposure in images. This makes it easier for researchers, with little or no programming background, to implement ZSL effectively to obtain insightful analytics from digital media. Such analytics can assist researchers and policy makers to propose regulations that can prevent alcohol exposure and eventually prevent alcohol consumption.

How to apply zero‐shot learning to text data in substance use research: An overview and tutorial with media data

“…The preliminary evidence to date suggests that, using ZSL, large language models produced relatively accurate results relative to human coders and task‐specific deep‐learning algorithms. In a recent study concerning identifying alcoholic beverages in images [30], we compared the ZSL performance of CLIP (contrastive language image pre‐training; Radford et al . [31]) to ABIDLA2 (alcoholic beverage identification deep‐learning algorithm, version 2), which is a task‐specific algorithm trained to detect alcoholic beverages in images [32].…”

“…ABIDLA2 was trained on more than 160 000 alcohol images and took months for a team of annotators to annotate the data and a large amount of computer resources to train and refine the algorithm. Despite this, we found that CLIP achieved comparable performance in identifying alcoholic beverages in images, and in some tasks performed better than ABIDLA2 [30]. Considering text analysis applications, ZSL large language models have achieved high accuracy for non‐substance use areas in tasks such as processing clinical text [33], classifying the topics of Tweets or news stories [34] and cross‐lingual sentiment classification [35].…”

“…), sentiment analyses or determining whether a post is pro or anti a specific policy. For example, in our research aiming to identify alcoholic beverages from images, we used ZSL—testing the use of both descriptive and name‐based phrases—and compared it to a dedicated supervised deep‐learning algorithm (ABIDLA2) to identify what beverage was presented [30]. Although the dedicated algorithm was more accurate at the finer‐grained classification tasks, such as distinguishing beer bottle versus beer can (ZSL using descriptive phrases = 73.88%; ABIDLA2 = 79.46%), ZSL using descriptive phrases achieved comparable recall accuracy when classifying more general categories (e.g.…”

“…‘this is a picture of someone drinking a beer’) improved accuracy relative to shorter phrases (e.g. ‘this is a beer’) [30] and we reported the results from both short phrases and longer phrases in the paper.…”

“…Some models may also struggle with ‘other’ or ‘neither’ categories. For example, we found that the phrase ‘other’ was not accurate in categorizing some non‐alcohol‐related images [30], probably because an image of a person drinking a bottle of water is closer to the phrase ‘a person drinking from a beer bottle’ than ‘other’. Therefore, instead of ‘other’, we included a more extensive list of phrases (e.g.…”