nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Models

Abstract

A rigorous formalization of desired system requirements is indispensable when performing any verification task. This often limits the application of verification techniques, as writing formal specifications is an error-prone and time-consuming manual task. To facilitate this, we present nl2spec, a framework for applying Large Language Models (LLMs) to derive formal specifications (in temporal logics) from unstructured natural language. In particular, we introduce a new methodology to detect and resolve the inherent ambiguity of system requirements in natural language: we utilize LLMs to map subformulas of the formalization back to the corresponding natural language fragments of the input. Users iteratively add, delete, and edit these sub-translations to amend erroneous formalizations, which is easier than manually redrafting the entire formalization. The framework is agnostic to specific application domains and can be extended to similar specification languages and new neural models. We perform a user study to obtain a challenging dataset, which we use to run experiments on the quality of translations. We provide an open-source implementation, including a web-based frontend.

Toward Auto-Modeling of Formal Verification for NextG Protocols: A Multimodal Cross- and Self-Attention Large Language Model Approach

“…LLMs Based Formal Verification: LLMs have demonstrated impressive reasoning and assertion capabilities for formal verification [9], [18], [19]. Research in [8], [9] has explored using LLMs to generate temporal logic specifi-cations and assertions from unstructured natural languages.…”

“…LLMs Based Formal Verification: LLMs have demonstrated impressive reasoning and assertion capabilities for formal verification [9], [18], [19]. Research in [8], [9] has explored using LLMs to generate temporal logic specifi-cations and assertions from unstructured natural languages. Meanwhile, studies in [20], [21] focus on leveraging LLMs to enhance BMC for identifying software vulnerabilities and deriving counterexamples.…”

“…In the past several months, both academic and industry sectors have increasingly focused on applying LLMs in the realm of formal verification. Two major areas of this application are using prompt engineering and LLMs for hardware assertion [8]- [10] and software system Bounded Model Checking (BMC) [8], [9]. To our knowledge, no published work yet explores the use of LLMs in verifying communicationrelated protocols and specifications, especially for the large and complex protocols of 3GPP releases.…”