Fully body visual self-modeling of robot morphologies

Abstract

Internal computational models of physical bodies are fundamental to the ability of robots and animals alike to plan and control their actions. These “self-models” allow robots to consider outcomes of multiple possible future actions without trying them out in physical reality. Recent progress in fully data-driven self-modeling has enabled machines to learn their own forward kinematics directly from task-agnostic interaction data. However, forward kinematic models can only predict limited aspects of the morphology, such as the position of end effectors or velocity of joints and masses. A key challenge is to model the entire morphology and kinematics without prior knowledge of what aspects of the morphology will be relevant to future tasks. Here, we propose that instead of directly modeling forward kinematics, a more useful form of self-modeling is one that could answer space occupancy queries, conditioned on the robot’s state. Such query-driven self-models are continuous in the spatial domain, memory efficient, fully differentiable, and kinematic aware and can be used across a broader range of tasks. In physical experiments, we demonstrate how a visual self-model is accurate to about 1% of the workspace, enabling the robot to perform various motion planning and control tasks. Visual self-modeling can also allow the robot to detect, localize, and recover from real-world damage, leading to improved machine resiliency.

Machine learning–driven self-discovery of the robot body morphology

“…14 represents a valid translation, we introduce a constraint and define the estimated JCP as follows: The estimated JCP of the jth joint is a point rj along the joint axis ζ that is a valid solution to Eq. 14 Because the actual angular acceleration ̇ in Eq. 14 is not measured, numerical differentiation of the IMU-measured angular velocity (denoted ̂ ) is used instead.…”

“…Most of the methods in the research literature to learn and adapt the body schema of a robot rely on a visual input source from which the robot can build an understanding of its geometry (9)(10)(11)(12)(13)(14). In contrast, complementary works have explored alternatives to extract body knowledge using the robot's perceptual apparatus to find internal representations (15) and to find and encode sensorimotor maps (16).…”

A Multi-Industry Analysis of the Future Use of AI Chatbots

“…Such AI is used in various functions, from voice recognition software like Siri and Alexa to medical diagnosis through pattern recognition techniques [14,15]. Other examples of current AI include FN Meka, an AI rapper signed to Capitol Records and research at Columbia University that taught a robot to visualise itself through AI [16]. Worldwide, governments and organisations are developing frameworks for the development and enactment of AI in daily life.…”

Neural Contact Fields: Tracking Extrinsic Contact with Tactile Sensing

“…With such advantages, neural fields are increasingly being explored and built on for robotics applications like learning policies [27], robot self-models for space occupancy queries [28], room scale online signed distance fields for navigation [29], and learning transformations from demonstrations of a pick-and-place task [30].…”