On Testing Constraint Programs

Abstract

The success of several constraint-based modeling languages such as OPL, ZINC, or COMET, appeals for better software engineering practices, particularly in the testing phase. This paper introduces a testing framework enabling automated test case generation for constraint programming. We propose a general framework of constraint program development which supposes that a first declarative and simple constraint model is available from the problem specifications analysis. Then, this model is refined using classical techniques such as constraint reformulation, surrogate and global constraint addition, or symmetrybreaking to form an improved constraint model that must be thoroughly tested before being used to address real-sized problems. We think that most of the faults are introduced in this refinement step and propose a process which takes the first declarative model as an oracle for detecting non-conformities. We derive practical test purposes from this process to generate automatically test data that exhibit non-conformities. We implemented this approach in a new tool called CPTEST that was used to automatically detect non-conformities on two classical benchmark programs, namely the Golomb rulers and the car-sequencing problem.

Fault Localization in Constraint Programs

“…As any other programs, constraint programs written in high-level modeling languages must be thoroughly verified before being used on real-size instances of satisfaction or optimization problems. In [13], we introduced a testing framework where a first highly declarative constraint model is took as a reference to detect non-conformities within a refined and optimized constraint program solving the same problem. These non-conformities result from faults introduced during the refinement process, coming either from the absence or the bad formulation of constraints.…”

“…For example, consider the N-queens problem solved by the OPL program of Fig.1. When N = 30, using a simple declarative model of N-queens, the testing framework of [13] reports a non-conformity: [30 29 28 ...1]. Although this vector indeed solves all the constraints of Fig.1, it shows a fault in the OPL program which obviously places many queens on the same diagonal.…”

“…The rest of the paper is organized as follows: Section II gives an overview of the constraint programs testing framework of [13] and introduces some notations. Section III presents our automatic fault localization algorithm.…”

“…We think that most of the faults are introduced in this refinement step. In our previous work [13], we introduced a testing framework taking the first declarative model as an oracle for detecting non-conformities in the optimized model. We defined various formal conformity relations handling the cases where a single solution is found, all the solutions are found or a cost function has to be optimized.…”

“…Detecting and localizing faults is therefore of great importance to lower the cost of the development of these constraint programs. In a previous work, we introduced a testing framework called CPTEST enabling automated test case generation for detecting non-conformities [13]. In this paper, we enhance this framework to introduce automatic fault localization in constraint programs.…”