Interactive synthesis of temporal specifications from examples and natural language

Gavran, Ivan; Darulová, Eva; Majumdar, Rupak

doi:10.1145/3428269

Cited by 12 publications

(5 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A survey of earlier research before the advent of deep learning is provided in [5]. Other approaches include an interactive method using SMT solving and semantic parsing [16], or structured temporal aspects in grounded robotics [46] and planning [33]. Neural networks have only recently being used to translate into temporal logics, e.g., by training a model for STL from scratch [22], fine-tuning language models [20], or an approach to apply GPT-3 [14,30] in a one-shot fashion, where [14] output a restricted set of declare templates [34] that can be translated to a fragment of LTLf [10].…”

Section: Natural Language To Linear-time Temporal Logicmentioning

confidence: 99%

nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Models

Matthias¹,

Hahn²,

Mendoza³

et al. 2023

Preprint

View full text Add to dashboard Cite

A rigorous formalization of desired system requirements is indispensable when performing any verification task. This often limits the application of verification techniques, as writing formal specifications is an error-prone and time-consuming manual task. To facilitate this, we present nl2spec, a framework for applying Large Language Models (LLMs) to derive formal specifications (in temporal logics) from unstructured natural language. In particular, we introduce a new methodology to detect and resolve the inherent ambiguity of system requirements in natural language: we utilize LLMs to map subformulas of the formalization back to the corresponding natural language fragments of the input. Users iteratively add, delete, and edit these sub-translations to amend erroneous formalizations, which is easier than manually redrafting the entire formalization. The framework is agnostic to specific application domains and can be extended to similar specification languages and new neural models. We perform a user study to obtain a challenging dataset, which we use to run experiments on the quality of translations. We provide an open-source implementation, including a web-based frontend.

show abstract

Section: Natural Language To Linear-time Temporal Logicmentioning

confidence: 99%

nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Models

Matthias¹,

Hahn²,

Mendoza³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…These learning algorithms have been applied to robotics. For example, [18]- [20] use the learning algorithm from [1], [2].…”

Section: I R E L At E D W O R Kmentioning

confidence: 99%

Learning Linear Temporal Properties for Autonomous Robotic Systems

Ghiorzi

Colledanchise

Piquet

et al. 2023

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

The problem of passive learning of linear temporal logic formulae consists in finding the best explanation for how two sets of execution traces differ, in the form of the shortest formula that separates the two sets. We approach the problem by implementing an exhaustive search algorithm optimized for execution speed. We apply it to the use-case of a robot moving in an unstructured environment as its battery discharges, both in simulation and in the real world. The results of our experiments confirm that our approach can learn temporal formulas explaining task failures in a case of practical interest.

show abstract

“…All approaches are typically domain-specific, where we provide a most general approach exploiting the generalization capabilities of language models. Gavran et al (2020) present an interactive method for synthesizing LTL specifications from example traces and natural language by combining SMT solving and semantic parsing.…”

Section: Related Workmentioning

confidence: 99%

Formal Specifications from Natural Language

Hahn¹,

Schmitt²,

Tillman³

et al. 2022

Preprint

View full text Add to dashboard Cite

We study the ability of language models to translate natural language into formal specifications with complex semantics. In particular, we fine-tune off-the-shelf language models on three datasets consisting of structured English sentences and their corresponding formal representation: 1) First-order logic (FOL), commonly used in software verification and theorem proving; 2) linear-time temporal logic (LTL), which forms the basis for industrial hardware specification languages; and 3) regular expressions (regex), frequently used in programming and search. Our experiments show that, in these diverse domains, the language models achieve competitive performance to the respective state-of-the-art with the benefits of being easy to access, cheap to fine-tune, and without a particular need for domainspecific reasoning. Additionally, we show that the language models have a unique selling point: they benefit from their generalization capabilities from pre-trained knowledge on natural language, e.g., to generalize to unseen variable names.

show abstract

Interactive synthesis of temporal specifications from examples and natural language

Cited by 12 publications

References 53 publications

nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Models

nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Models

Learning Linear Temporal Properties for Autonomous Robotic Systems

Formal Specifications from Natural Language

Contact Info

Product

Resources

About