On the Expressivity of Markov Reward (Extended Abstract)

Abstract

Modern SAT solvers are based on a paradigm named conflict driven clause learning (CDCL), while local search is an important alternative. Although there have been attempts combining these two methods, this work proposes deeper cooperation techniques. First, we relax the CDCL framework by extending promising branches to complete assignments and calling a local search solver to search for a model nearby. More importantly, the local search assignments and the conflict frequency of variables in local search are exploited in the phase selection and branching heuristics of CDCL. We use our techniques to improve three typical CDCL solvers (glucose, MapleLCMDistChronoBT and Kissat). Experiments on benchmarks from the Main tracks of SAT Competitions 2017-2020 and a real world benchmark of spectrum allocation show that the techniques bring significant improvements, particularly on satisfiable instances. A resulting solver won the Main Track SAT category in SAT Competition 2020 and also performs very well on the spectrum allocation benchmark. As far as we know, this is the first work that meets the standard of the challenge ``Demonstrate the successful combination of stochastic search and systematic search techniques, by the creation of a new algorithm that outperforms the best previous examples of both approaches.'' (AAAI 1997) on standard application benchmarks.

Policy Optimization with Linear Temporal Logic Constraints

“…For the two policies, the avg. costs are g(1) = R and g(2) = pS + (1 − p)R. Strangely, we must set R > S in order for g(2) < g (1).…”

“…In other words, probability-optimal policies are those that satisfy the entirety of the task, both desired and required behaviors, whereas V P π,λ ≡ (J π + λg π )P[π |= ϕ] is the normalized value function 1 , corresponding to a notion of energy or effort required, with λ representing the tradeoff between gain and transient cost. We will often omit the dependence of V on P and λ for brevity.…”

“…Hence, the SSP guarantee (our center term) and the standard RL guarantee are very similar. The first term 1 β does not appear in standard RL literature because there is no constraint verification needed, but in practice will be dominated by the other terms. The last term is also similar to the center term.…”

“…, applying union bound over all (s, a, s ) ∈ S × A × S, and observing that Z i ∼ P (s, a, s ) is a Bernoulli random variable with empirical variance Vn = P (s, a, s )(1 − P (s, a, s )) yields the result: {∀s, a, s ∈ S × A × S, ∀n > 1 : |P (s, a, s ) − P (s, a, s )| ≤ ψ sas (n)} holds with prob 1 − δ Observing that ψ sas (n) ≤ ψ(n) for all n > 1 because ψ sas (n) takes on a maximum when P (s, a, s ) = 1 2 , completes the proof. Lemma B.2 (Inverting E).…”

“…By combining these objectives into scalar costs, one erases the distinction between these two categories of tasks. Also, there is recent theoretical evidence that certain tasks are simply not reducible to scalar costs [1] (see Section 2). In practice, one circumvents these challenges using heuristics such as adding "breadcrumbs" [52].…”