The 3rd Reactive Synthesis Competition (SYNTCOMP 2016): Benchmarks, Participants & Results

Abstract

We report on the benchmarks, participants and results of the third reactive synthesis competition (SYNTCOMP 2016). The benchmark library of SYNTCOMP 2016 has been extended to benchmarks in the new LTL-based temporal logic synthesis format (TLSF), and 2 new sets of benchmarks for the existing AIGER-based format for safety specifications. The participants of SYNTCOMP 2016 can be separated according to these two classes of specifications, and we give an overview of the 6 tools that entered the competition in the AIGER-based track, and the 3 participants that entered the TLSF-based track. We briefly describe the benchmark selection, evaluation scheme and the experimental setup of SYNTCOMP 2016. Finally, we present and analyze the results of our experimental evaluation, including a comparison to participants of previous competitions and a legacy tool. IntroductionSince the definition of the problem more than 50 years ago [16], the automatic synthesis of reactive systems from formal specifications is one of the major challenges of computer science. Research into the basic questions related to the problem has led to a large body of theoretical results, but their impact on the practice of system design has been rather limited. To increase the impact of theoretical advancements in synthesis, the reactive synthesis competition (SYNTCOMP) has been founded in 2014 [27]. The competition is designed to foster research in scalable and user-friendly implementations of synthesis techniques by establishing a standard benchmark format, maintaining a challenging public benchmark library, and providing a dedicated and independent platform for the comparison of tools under consistent experimental conditions.The venues. A design choice for the first two competitions was to focus on safety properties specified as monitor circuits in an extension of the AIGER format known from the hardware model checking competition [15,25]. SYNTCOMP 2016 introduces the first major extension of the competition: in addition to the existing competition track, we introduce a new track that is based on properties in full linear temporal logic (LTL), given in the temporal logic synthesis format (TLSF) recently introduced by Jacobs, Klein and Schirmer [29].The organization team of SYNTCOMP 2016 consisted of R. Bloem and S. Jacobs, with technical assistance from J. Kreber for the setup and execution, and from F. Klein for the integration of TLSF.The rest of this paper describes the design, benchmarks, participants, and results of SYNTCOMP 2016. We present the benchmark set for SYNTCOMP 2016 in Section 2, followed by a description of the setup, rules and execution of the competition in Section 3. In Section 4 we give an overview of the participants of SYNTCOMP 2016, focusing on changes compared to last year's participants. Finally, the experimental results are presented and analyzed in Section 5.Note that more details on the goals and design of the competition can be found in the sister paper that discusses the design of SYNTCOMP in 2016 and the future [26]. Be...

The 4th Reactive Synthesis Competition (SYNTCOMP 2017): Benchmarks, Participants & Results

“…We describe the implemented techniques and optimizations of the new tools, followed by a brief overview of the updated tools. For additional details on the latter, we refer to the report of SYNTCOMP 2016 [23].…”

“…We start by describing two new benchmark classes in TLSF, followed by a listing of the classes of benchmarks (in both TLSF and AIGER format) that have already been used in previous competitions. For more details on these existing benchmarks, we refer to the previous competition reports [22][23][24].…”

“…For more details consult the report on SYNTCOMP 2016 [23] and the original sources. The existing benchmark library consists of the following classes of benchmarks:…”

“…For additional details on the implemented techniques and optimizations, we refer to the previous SYNTCOMP reports [22][23][24].…”

“…[22][23][24] A design choice for the first two competitions was to focus on safety properties specified as monitor circuits in an extension of the AIGER format known from the hardware model checking competition [9,20]. SYNTCOMP 2016 introduced the first major extension of the competition by adding a new track that is based on properties in full linear temporal logic (LTL), given in the temporal logic synthesis format (TLSF) [21,25].…”