Learning to Associate Words and Images Using a Large-Scale Graph

Abstract

We develop an approach for unsupervised learning of associations between co-occurring perceptual events using a large graph. We applied this approach to successfully solve the image captcha of China's railroad system. The approach is based on the principle of suspicious coincidence, originally proposed by Barlow [1], who argued that the brain builds a statistical model of the world by learning associations between events that repeatedly co-occur. In this particular problem, a user is presented with a deformed picture of a Chinese phrase and eight low-resolution images. They must quickly select the relevant images in order to purchase their train tickets. This problem presents several challenges: (1) the teaching labels for both the Chinese phrases and the images were not available for supervised learning, (2) no pre-trained deep convolutional neural networks are available for recognizing these Chinese phrases or the presented images, and (3) each captcha must be solved within a few seconds. We collected 2.6 million captchas, with 2.6 million deformed Chinese phrases and over 21 million images. From these data, we constructed an association graph, composed of over 6 million vertices, and linked these vertices based on co-occurrence information and feature similarity between pairs of images. We then trained a deep convolutional neural network to learn a projection of the Chinese phrases onto a 230dimensional latent space. Using label propagation, we computed the likelihood of each of the eight images conditioned on the latent space projection of the deformed phrase for each captcha. The resulting system solved captchas with 77% accuracy in 2 seconds on average. Our work, in answering this practical challenge, illustrates the power of this class of unsupervised association learning techniques, which may be related to the brain's general strategy for associating language stimuli with visual objects on the principle of suspicious coincidence.

Towards understanding the security of modern image captchas and underground captcha-solving services

“…Note that for selection-based captchas, we also compare SelAttack with two existing state-of-the-art attack methods [5,6] . The evaluation results suggest that SelAttack is more effective, both in terms of success rate and speed (the time of solving a captcha challenge).…”

“…Sivakorn et al [5] Ya et al [6] This paper (e.g., rotation and distortion) being proposed, along with design guidelines and suggestions for increased security [7][8][9][10][11][12] . The security of image captchas, on the other hand, is still in need of more comprehensive study.…”

“…The security of image captchas, on the other hand, is still in need of more comprehensive study. Specifically, existing works either focus on synthetic image captchas [13,14] , or on some particular cases of captcha schemes (e.g., ReCaptcha 2015) [5,6] .…”

“…For example, in 2016, Sivakorn et al [5] utilized deep learning techniques to solve one of the most popular image captcha schemes, ReCaptcha. Ya et al [6] applied advanced vision techniques to break image captchas from 12306.cn. Also, many commercial companies have deployed powerful online services to undertake vision tasks, such as image classification and object detection, and these services can be maliciously used by abusive programs to break image captchas.…”

“…Most recently, Sivakorn et al [5] designed a novel attack on ReCaptcha that leveraged deep learning techniques. Ya et al [6] also developed a novel learning approach in constructing a large association graph, and then applied this graph to break captchas from 12306.cn.…”