Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformers

Abstract

Natural language is the most intuitive medium for us to interact with other people when expressing commands and instructions. However, using language is seldom an easy task when humans need to express their intent towards robots, since most of the current language interfaces require rigid templates with a static set of action targets and commands. In this work, we provide a flexible languagebased interface for human-robot collaboration, which allows a user to reshape existing trajectories for an autonomous agent. We take advantage of recent advancements in the field of large language models (BERT and CLIP) to encode the user command, and then combine these features with trajectory information using multi-modal attention transformers. We train the model using imitation learning over a dataset containing robot trajectories modified by language commands, and treat the trajectory generation process as a sequence prediction problem, analogously to how language generation architectures operate. We evaluate the system in multiple simulated trajectory scenarios, and show a significant performance increase of our model over baseline approaches. In addition, our realworld experiments with a robot arm show that users significantly prefer our natural language interface over traditional methods such as kinesthetic teaching or cost-function programming. Our study shows how the field of robotics can take advantage of large pre-trained language models towards creating more intuitive interfaces between robots and machines. Project webpage: https://arthurfenderbucker. github.io/NL_trajectory_reshaper/

LATTE: LAnguage Trajectory TransformEr

“…By computing the cosine similarity vector s between the embedding we are identify a possible target object that the user is referring to. In the section 3.1, we show that using the object's images or object's names (as done in [29]) brings equivalent results, since CLIP maps both images and text to a joint latent space. Finally we concatenate the similarity vector s and the semantic features q BERT (z in |L in ) forming what we call features embedding q F .…”

“…Particularly highlight should also be given to the works [28,29], which are more closely related to our approach. In [28], it is proposed a method of mapping NL to transformations of cost functions.…”

“…Although it is able to modify the robot behavior using language, this approach is still restricted for 2D scenarios and cost based motion planner. The work in [29] proposed a transformer based approach for 2D trajectory reshaping on using NL. User studies validated the effectiveness of a task adaptation scheme through NL in compare to other types of human robot interaction through user studies.…”

Reconstructing Action-Conditioned Human-Object Interactions Using Commonsense Knowledge Priors

“…visual relationships [19] Language is also used as an additional input to guide tasks such as video summarization [22]. In robotics or policy learning, the agents not only follow instructions, but also learn to update semantic map for robot manipulation [24], trajectory reshaping [5], and new skills with language inputs [32].…”

Adversarial Attacks on Optimization based Planners

“…Transformers in robotics: Transformers were originally introduced in the language processing domain [120], but quickly proved to be useful in modeling long-range data dependencies other domains. Within robotics we see the first transformers architectures being used for trajectory forecasting [121], motion planning [122,123], and reinforcement learning [124,125]. The main difference between these works and GRID is that they are focused on training a model for a single task, while we propose learning representations amenable to multiple downstream tasks for a robot.…”