Beyond Planar Symmetry: Modeling Human Perception of Reflection and Rotation Symmetries in the Wild

Abstract

Humans take advantage of real world symmetries for various tasks, yet capturing their superb symmetry perception mechanism with a computational model remains elusive. Motivated by a new study demonstrating the extremely high inter-person accuracy of human perceived symmetries in the wild, we have constructed the first deeplearning neural network for reflection and rotation symmetry detection (Sym-NET), trained on photos from MS-COCO (Microsoft-Common Object in COntext) dataset with nearly 11K consistent symmetry-labels from more than 400 human observers. We employ novel methods to convert discrete human labels into symmetry heatmaps, capture symmetry densely in an image and quantitatively evaluate Sym-NET against multiple existing computer vision algorithms. On CVPR 2013 symmetry competition testsets and unseen MS-COCO photos, Sym-NET significantly outperforms all other competitors. Beyond mathematically well-defined symmetries on a plane, Sym-NET demonstrates abilities to identify viewpoint-varied 3D symmetries, partially occluded symmetrical objects, and symmetries at a semantic level.

Pros and cons of GAN evaluation measures

“…In this regard, borrowing from other fields such as natural scene statistics and cognitive vision can be rewarding. For example, understanding how humans perceive symmetry [134,135] or image clutter [136] in generated images versus natural scenes can give clues regarding the plausibility of the generated images.…”

Estimating Efforts and Success of Symmetry-Seeing Machines by Use of Synthetic Data

“…The world under discussion is the world of pictures. In contrast to many machine-vision approaches from Grenander's Pattern Theory [6] to modern convolutional deep learning models [1] where the input is a pixel matrix, this work prefers a continuous domain, and accordingly sparse representations. Among the possible two-dimensional manifolds, the Cartesian vector space over the field of real numbers is the preferred choice.…”

“…The objects themselves are not represented at all. Funk and Liu use such rationale for their approach [1], which sets the current best performance in recognizing reflection and rotational symmetry. In their discussion, they use the dichotomy dense-sparse coding, where dense refers to approaches that cover the complete domain by sampling it with activation cells, or "neural layers".…”

“…Estimating the recognition success of symmetry-seeing machines as well as comparing or assessing their performance requires a representative set of images together with ground truth on the contained symmetry. Funk and Liu proposed to use heatmaps as ground truth [1]. The heatmap has the same raster format as the corresponding image, and it contains values between 0 and 1.…”

“…Zero means there is no symmetry at that location, and 1 means there is maximal symmetry there. A separate map is needed for each type of symmetry, e.g., in Reference [1] one map per image codes reflection axes, and one map per image codes rotation centers. This paper contains various such heatmaps using the inverted graphical convention that "hot" or "1.0" is indicated black, and "cold" or "0.0" is indicated as white, since white is less salient on white paper.…”