Hybrid Classification and Reasoning for Image-Based Constraint Solving

Abstract

There is an increased interest in solving complex constrained problems where part of the input is not given as facts, but received as raw sensor data such as images or speech. We will use 'visual sudoku' as a prototype problem, where the given cell digits are handwritten and provided as an image thereof. In this case, one first has to train and use a classifier to label the images, so that the labels can be used for solving the problem. In this paper, we explore the hybridisation of classifying the images with the reasoning of a constraint solver. We show that pure constraint reasoning on predictions does not give satisfactory results. Instead, we explore the possibilities of a tighter integration, by exposing the probabilistic estimates of the classifier to the constraint solver. This allows joint inference on these probabilistic estimates, where we use the solver to find the maximum likelihood solution. We explore the tradeoff between the power of the classifier and the power of the constraint reasoning, as well as further integration through the additional use of structural knowledge. Furthermore, we investigate the effect of calibration of the probabilistic estimates on the reasoning. Our results show that such hybrid approaches vastly outperform a separate approach, which encourages a further integration of prediction (probabilities) and constraint solving.

Sudoku Assistant – an AI-Powered App to Help Solve Pen-and-Paper Sudokus

“…All rights reserved. Better predictions with a hybrid of prediction and reasoning Therefore, we proposed an alternative approach in (Mulamba et al 2020), which features a deeper integration of the digit classification and the reasoning required to solve the Sudoku puzzle. This approach is schematically shown in Fig.…”

“…However this requires solver-specific mechanisms whereas our approach works with any constraint solver. Another limitation, also present in (Bai, Chen, and Gomes 2021;Mulamba et al 2020) is that instances are built by sampling images from MNIST (Deng 2012), on which recent machine learning approaches achieve near-perfect accuracy (An et al 2020). Images from a phone camera are in RGB space, can contain artifacts such as grid borders and are overall noisier depending on the angle of the camera or lighting conditions.…”