Metamorphic Testing of Constraint Solvers

Abstract

Abstract. Constraint solvers are complex pieces of software and are notoriously difficult to debug. In large part this is due to the difficulty of pinpointing the source of an error in the vast searches these solvers perform, since the effect of an error may only come to light long after the error is made. In addition, an error does not necessarily lead to the wrong result, further complicating the debugging process. A major source of errors in a constraint solver is the complex constraint propagation algorithms that provide the inference that controls and directs the search. In this paper we show that metamorphic testing is a principled way to test constraint solvers by comparing two different implementations of the same constraint. Specifically, specialised propagators for the constraint are tested against the general purpose table constraint propagator. We report on metamorphic testing of the constraint solver Minion. We demonstrate that the metamorphic testing method is very effective for finding artificial bugs introduced by random code mutation.

Adaptive metamorphic testing with contextual bandits

“…Metamorphic Testing (MT) has been applied to a variety of domains and applications, see [24,2] for an in-depth overview. Successful application domains include testing of driverless cars [7], search engines [8], machine translation systems [25], performance testing [26,27], constraint solvers [28] or bioinformatics [9]. Previous works already focused on its automation, for example by exploring algorithms to specifically identify fault-revealing inputs [29] or performing an empirical study on selecting good Metamorphic Relations (MRs) [30].…”

Certified Symmetry and Dominance Breaking for Combinatorial Optimisation

“…Despite these difficulties, symmetry elimination using both manual and automatic techniques has been key to many successes across modern combinatorial optimisation paradigms such as constraint programming (CP) (Garcia de la Banda et al 2014), Boolean satisfiability (SAT) (Biere et al 2021), and mixed-integer programming (MIP) (Achterberg and Wunderling 2013). As these optimisation technologies are increasingly being used for high-value and life-affecting decision-making processes, it becomes vital that we can trust their outputs-and unfortunately, current solvers do not always produce the correct answer (Brummayer, Lonsing, and Biere 2010;Cook et al 2013;Akgün et al 2018;Gillard, Schaus, and Deville 2019). The most promising way to address this problem appears to be to use certification, or proof logging, where a solver must produce an efficiently machine-verifiable certificate that the solution given is correct (Alkassar et al 2011;McConnell et al 2011).…”

QMaxSATpb: A Certified MaxSAT Solver

“…Since some of these applications involve high-value and life-affecting decisionmaking processes (e.g., verifying software that drives our transportation infrastructure [44], or matching donors and recipients for Kidney transplants [34]), it is of utmost importance that the answers produced by the solvers be completely reliable. Unfortunately, the reality is different: the constant need for more efficient and advanced algorithms forms an excellent breeding ground for bugs, resulting in numerous reports of solvers outputting faulty answers [9,11,1,20].…”