Towards Visual Semantics

Abstract

Lexical Semantics is concerned with how words encode mental representations of the world, i.e., concepts. We call this type of concepts, classification concepts. In this paper, we focus on Visual Semantics, namely, on how humans build concepts representing what they perceive visually. We call this second type of concepts, substance concepts. As shown in the paper, these two types of concepts are different and, furthermore, the mapping between them is many-to-many. In this paper we provide a theory and an algorithm for how to build substance concepts which are in a one-to-one correspondence with classifications concepts, thus paving the way to the seamless integration between natural language descriptions and visual perception. This work builds upon three main intuitions: (i) substance concepts are modeled as visual objects, namely, sequences of similar frames, as perceived in multiple encounters; (ii) substance concepts are organized into a visual subsumption hierarchy based on the notions of and ; (iii) the human feedback is exploited not to name objects, but, rather, to align the hierarchy of substance concepts with that of classification concepts. The learning algorithm is implemented for the base case of a hierarchy of depth two. The experiments, though preliminary, show that the algorithm manages to acquire the notions of and with reasonable accuracy, this despite seeing a small number of examples and receiving supervision on a fraction of them.

Visual Ground Truth Construction as Faceted Classification

“…On the other hand, by image classification, we mean the activity which determines what object classes are present in an image (Russakovsky et al 2015). This latter task, which is semantics-intensive, is performed post object detection, wherein, given a continuous feed of images, the goal is to organize the objects in them into a classification hierarchy of visual objects based on their visual genus-differentia (Giunchiglia, Erculiani, and Passerini 2021). We ground the above ordered distinction in the empirically validated theory of teleosemantics (Giunchiglia and Fumagalli 2016;Giunchiglia and Fumagalli 2017;Millikan 2020), as originally postulated by the philosopher Ruth Millikan (Millikan 1989;Millikan 2000;Millikan 2004;Millikan 2005).…”

“…This process involves (continual) extraction of distinguishing visual properties unique to a substance concept, and thereby delineating substance concepts, for instance, via bounding polygons. Subsequently, substance concepts are stored in a "cumulative memory M of all the times they were previously perceived" (Giunchiglia, Erculiani, and Passerini 2021).…”

“…Secondly, the key to continual detection of a substance as different from substance concepts is grounded in its internal causal factor (Giunchiglia and Fumagalli 2016) which is incrementally manifested and extracted as perceivable visual properties. Finally, and most importantly, substance concepts are perdurant representations (Gangemi et al 2002) of objects as "we never have a full (visual) picture of the object but that its visual representation is built progressively, in time" (Giunchiglia, Erculiani, and Passerini 2021). The perdurant representation is, in fact, grounded in the notions of space and time persistency (Giunchiglia, Erculiani, and Passerini 2021) which ensures that even if an object isn't fully visually perceivable at once, it does exhibit very slow spatiotemporal variance which allows for its detection and substance concept representation to be built, progressively and cumulatively, in time.…”

“…As a matter of fact, an early version of this type of problems was crystallized, already in 2000, as the Semantic Gap Problem (SGP) (Smeulders et al 2000), where the SGP was described as the "lack of coincidence between the information that one can extract from the visual data and the interpretation that the same data have for a user in a given situation". This problem, still unsolved, has been generalized in (Giunchiglia, Erculiani, and Passerini 2021) as the fact that there is a many-to-many mapping between the information extracted from the visual data and its possible contextual interpretations. As from this work, the SGP is a very general phenomenon which manifests itself anytime an object inside an image is linguistically annotated multiple times, possibly even by the same person.…”

“…The issue is to align this hierarchy with the visual semantics encoded in the objects depicted in images. To this extent we follow the theory introduced in (Giunchiglia, Erculiani, and Passerini 2021) and associate objects, as depicted in (multiple) images, with sets of visual properties, e.g., sets of (visual) frames, which describe their appearance. In this way, the annotation process is no longer that of annotating (objects in) images with labels but, rather, of aligning the visual properties of (objects in) images with those lexically described properties which describe the meaning of ImageNet classes, i.e., their definition, or gloss, at it is called in WordNet (Miller 1995).…”