Visual Information Theory and Visual Representation for Achieving Provable Bounds in Vision-Based Control and Decision

Soatto, Stefano

doi:10.21236/ada611903

Cited by 4 publications

(4 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that such objects often exhibit complex geometry, topology, and photometry, thus precluding the use of off-the-shelf laser scanners (due to specular reflections); volume displacement techniques, e.g., submerging objects in water, cannot be easily employed as many objects either float (e.g., apples), absorb water (e.g., cardboard packaging for foodstuffs, stuffed animals), or are permanently damaged by water (e.g., hand-held consumer electronics). Further, we wished to measure volume in a manner as analogous as possible to the way in which humans do so without access to haptic information, i.e., on the basis of visual information alone [ 24 ]. For example, visual inspection prior to lifting would provide no information about internal cavities (as in hollow or porous objects).…”

Section: Methodsmentioning

confidence: 99%

Smaller = Denser, and the Brain Knows It: Natural Statistics of Object Density Shape Weight Expectations

2015

View full text Add to dashboard Cite

If one nondescript object’s volume is twice that of another, is it necessarily twice as heavy? As larger objects are typically heavier than smaller ones, one might assume humans use such heuristics in preparing to lift novel objects if other informative cues (e.g., material, previous lifts) are unavailable. However, it is also known that humans are sensitive to statistical properties of our environments, and that such sensitivity can bias perception. Here we asked whether statistical regularities in properties of liftable, everyday objects would bias human observers’ predictions about objects’ weight relationships. We developed state-of-the-art computer vision techniques to precisely measure the volume of everyday objects, and also measured their weight. We discovered that for liftable man-made objects, “twice as large” doesn’t mean “twice as heavy”: Smaller objects are typically denser, following a power function of volume. Interestingly, this “smaller is denser” relationship does not hold for natural or unliftable objects, suggesting some ideal density range for objects designed to be lifted. We then asked human observers to predict weight relationships between novel objects without lifting them; crucially, these weight predictions quantitatively match typical weight relationships shown by similarly-sized objects in everyday environments. These results indicate that the human brain represents the statistics of everyday objects and that this representation can be quantitatively abstracted and applied to novel objects. Finally, that the brain possesses and can use precise knowledge of the nonlinear association between size and weight carries important implications for implementation of forward models of motor control in artificial systems.

show abstract

Section: Methodsmentioning

confidence: 99%

Smaller = Denser, and the Brain Knows It: Natural Statistics of Object Density Shape Weight Expectations

2015

View full text Add to dashboard Cite

show abstract

“…Much of the research on perception and representation learning, both in ML and neuroscience, has focused on object recognition. In ML, this line of research has historically emphasized the importance of learning representations that are invariant to transformations like pose or illumination (Lowe, 1999 ; Dalal and Triggs, 2005 ; Sundaramoorthi et al, 2009 ; Soatto, 2010 ; Krizhevsky et al, 2012 ). In this framework, transformations are considered nuisance variables to be thrown away ( Figures 1A , 2B ).…”

Section: What Are Symmetries?mentioning

confidence: 99%

Symmetry-Based Representations for Artificial and Biological General Intelligence

Higgins

Racanière

Rezende

2022

Front. Comput. Neurosci.

View full text Add to dashboard Cite

Biological intelligence is remarkable in its ability to produce complex behavior in many diverse situations through data efficient, generalizable, and transferable skill acquisition. It is believed that learning “good” sensory representations is important for enabling this, however there is little agreement as to what a good representation should look like. In this review article we are going to argue that symmetry transformations are a fundamental principle that can guide our search for what makes a good representation. The idea that there exist transformations (symmetries) that affect some aspects of the system but not others, and their relationship to conserved quantities has become central in modern physics, resulting in a more unified theoretical framework and even ability to predict the existence of new particles. Recently, symmetries have started to gain prominence in machine learning too, resulting in more data efficient and generalizable algorithms that can mimic some of the complex behaviors produced by biological intelligence. Finally, first demonstrations of the importance of symmetry transformations for representation learning in the brain are starting to arise in neuroscience. Taken together, the overwhelming positive effect that symmetries bring to these disciplines suggest that they may be an important general framework that determines the structure of the universe, constrains the nature of natural tasks and consequently shapes both biological and artificial intelligence.

show abstract

“…b) Task: From a given sample of data z, "accomplishing a task" [30] refers to the ability to infer the corresponding label y ∈ Y. While in other works [27]- [29] the Markov chain x → z → y reads data → representation → task, in our multi-sensor setting it refers to scene → data → task.…”

Section: Problem Formalizationmentioning

confidence: 99%

Wormhole Learning

Zanardi

Zilly

Aumiller

et al. 2019

2019 International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

Typically, to enlarge the operating domain of an object detector, more labeled training data is required. We describe a method called wormhole learning, which allows to extend the operating domain without additional data, but only with temporary access to an auxiliary sensor with certain invariance properties.We describe the instantiation of this principle with a regular visible-light RGB camera as the main sensor, and an infrared sensor as the temporary sensor. We start with a pre-trained RGB detector; then we train the infrared detector based on the RGB-inferred labels; finally we re-train the RGB detector based on the infrared-inferred labels. After these two transferlearning steps, the RGB detector has enlarged its operating domain by inheriting part of the invariance to illumination of the infrared sensor; in particular, the RGB detector is now able to see much better at night.We analyze the wormhole learning phenomenon by bounding the possible gain in accuracy using mutual information properties of the two sensors and considered operating domain.

show abstract

Visual Information Theory and Visual Representation for Achieving Provable Bounds in Vision-Based Control and Decision

Cited by 4 publications

References 11 publications

Smaller = Denser, and the Brain Knows It: Natural Statistics of Object Density Shape Weight Expectations

Smaller = Denser, and the Brain Knows It: Natural Statistics of Object Density Shape Weight Expectations

Symmetry-Based Representations for Artificial and Biological General Intelligence

Wormhole Learning

Contact Info

Product

Resources

About