Formal Specifications from Natural Language

Abstract

We study the ability of language models to translate natural language into formal specifications with complex semantics. In particular, we fine-tune off-the-shelf language models on three datasets consisting of structured English sentences and their corresponding formal representation: 1) First-order logic (FOL), commonly used in software verification and theorem proving; 2) linear-time temporal logic (LTL), which forms the basis for industrial hardware specification languages; and 3) regular expressions (regex), frequently used in programming and search. Our experiments show that, in these diverse domains, the language models achieve competitive performance to the respective state-of-the-art with the benefits of being easy to access, cheap to fine-tune, and without a particular need for domainspecific reasoning. Additionally, we show that the language models have a unique selling point: they benefit from their generalization capabilities from pre-trained knowledge on natural language, e.g., to generalize to unseen variable names.

NL2LTL – a Python Package for Converting Natural Language (NL) Instructions to Linear Temporal Logic (LTL) Formulas

“…Related Work Existing works fall under two buckets: one which admits support for a range of LTL formulas but compromises on the expressiveness of the input (Hahn et al 2022;Schmitt 2022;Narizzano et al 2018;Narizzano and Vuotto 2017), and the other which admits natural language inputs but were built for a particular domain like robotics and are not readily useful as a general purpose package for practitioners (Wang et al 2020;Wang 2020;Nikora and Balcom 2009;Dwyer, Avrunin, and Corbett 1998;Kim, Banks, and Shah 2017;Lignos et al 2015). Furthermore, among these works, other than (Wang 2020;Narizzano and Vuotto 2017;Schmitt 2022), none have publicly available code and are therefore not readily usable for practitioners.…”

nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Models

“…Other approaches include an interactive method using SMT solving and semantic parsing [16], or structured temporal aspects in grounded robotics [46] and planning [33]. Neural networks have only recently being used to translate into temporal logics, e.g., by training a model for STL from scratch [22], fine-tuning language models [20], or an approach to apply GPT-3 [14,30] in a one-shot fashion, where [14] output a restricted set of declare templates [34] that can be translated to a fragment of LTLf [10]. Translating natural langauge to LTL has especially been of interest to the robotics community (see [17] for an overview), where datasets and application domains are, in contrast to our setting, based on structured natural language.…”

“…LLMs are Transformers [43], which is the state of the art neural architecture for natural language proccessing. Additionally, Transformers have shown remarkable performance when being applied to classical problems in verification (e.g., [19,41,26,9]), reasoning (e.g., [28,51]), as well as the auto-formalization [36] of mathematics and formal specifications (e.g., [50,20,22]).…”

“…While fine-tuning neural models on a specific translation task remains a valid approach showing also initial success in generalizing to unstructured natural language when translating to LTL [20], a common technique to obtain high performance with limited amount of labeled data is so-called "few-shot prompting" [4]. The language model is presented a natural language description of the task usually accompanied with a few examples that demonstrate the input-output behavior.…”

“…This experiment demonstrates that the initial translations are of high quality, which is important to ensure an efficient workflow. We compared our approach to fine-tuning language models on structured data [20] and to an approach using GPT-3 or Rasa [3] to translate natural language into a restricted set of declare patterns [14] (which could not handle most of the instances in the benchmark data set, even when replacing the atomic propositions with their used entities). The results of evaluating the accuracy of the initial translations on our benchmark expert set is shown in Table 1.…”