Shellcode_IA32: A Dataset for Automatic Shellcode Generation

Abstract

We take the first step to address the task of automatically generating shellcodes, i.e., small pieces of code used as a payload in the exploitation of a software vulnerability, starting from natural language comments. We assemble and release a novel dataset (Shellcode IA32), consisting of challenging but common assembly instructions with their natural language descriptions. We experiment with standard methods in neural machine translation (NMT) to establish baseline performance levels on this task.

A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspective

“…BLEU is used to evaluate code generation systems since many prior works in code generation formulated the problem as a machine translation problem of translating English to code snippets (e.g. (Liguori et al, 2021a)). Both exact match and averaged token level BLEU scores have been extensively used in evaluating code generation models (Liguori et al, 2021a,b;Oda et al, 2015b;Ling et al, 2016;Gemmell et al, 2020).…”

“…Furthermore, code generation enables users to build code more effectively and efficiently and enhance the overall software engineering process. Code software engineering (Miltner et al, 2019;, robotics (Kuhlmann et al, 2004), and cyber-security (You et al, 2017;Liguori et al, 2021a;Frempong et al, 2021; (a) An assembly code generation task. The task is to generate the assembly code that is then compiled into shellcode (small pieces of code used as a payload to exploit software vulnerabilities) using the natural language descriptions on the right.…”

“…The dataset contains multi-line snippets mapping onto one intent. Lines 4-5, 6-7-8, 9-10, 11-12 are multi-line snippets (Liguori et al, 2021a).…”

“…Recent works in code generation have been used to assist data scientists to perform data visualization and file manipulation , generating bash commands (Lin et al, 2018), generate exploits (Liguori et al, 2021b,a;Frempong et al, 2021;Liguori et al, 2022), solve interviewlevel programming questions (Hendrycks et al, 2021), manipulate data , and generate code snippets from natural language descriptions in many programming languages. These programming languages include but are not limited to Python (Xu et al, 2020a;Ling et al, 2016;Liguori et al, 2021b), Java (Ling et al, 2016), SQL (Zhong et al, 2017), Excel macro commands (Gulwani and Marron, 2014), Assembly (Liguori et al, 2021a(Liguori et al, ,b, 2022, and JavaScript (Frempong et al, 2021).…”

“…Automatic exploit generation is defined as an offensive security technique in which software exploits are automatically generated to explore and test critical vulnerabilities before malicious attackers discover such vulnerabilities (Avgerinos et al, 2011). With the goal of offensive security in mind, Liguori et al (Liguori et al, 2021a) share a dataset, Shell-code_IA32, for shellcode generation from natural language descriptions. Shellcodes are compiled from assembly programs for the 32-bit version of the x86 Intel Architecture and contain a payload that is used in exploiting software vulnerabilities.…”