RT-1: Robotics Transformer for Real-World Control at Scale

Brohan, Anthony; Brown, Noah A.; Justice, Carbajal,; Chebotar, Yevgen; Dabis, Joseph; Finn, Chelsea; Gopalakrishnan, Keerthana; Hausman, Karol; Herzog, Alex; Hsu, Jasmine; Ibarz, Julian; Ichter, Brian; Irpan, Alex; Jackson, Tomas; Jesmonth, Sally; Joshi, Nikhil J; Julian, Ryan R.; Kalashnikov, Dmitry; Yuheng, Kuang,; Leal, Isabel; Lee, Kuang-Huei; Levine, Sergey; Lu, Yao; Malla, Utsav; Manjunath, Deeksha; Mordatch, Igor; Nachum, Ofir; Parada, Carolina; Peralta, Jodilyn; Perez, Emily; Pertsch, Karl; Quiambao, Jornell; Rao, Kanishka; Ryoo, Michael S.; Salazar, Grecia; Sanketi, Pannag R.; Sayed, Kevin; Jaspiar, Singh,; Stone, Austin V.; Tan, Clayton; Tran, H. L.; Vanhoucke, Vincent; Vega, Steve; Vuong, Quan V.; Xia, Fei; Ted, Xiao,; Xu, Ping; Sichun, Xu,; Yu, Tong; Zitkovich, Brianna

doi:10.48550/arxiv.2212.06817

Cited by 37 publications

(70 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Expanding the RL-controller's policy to include a diverse set of tasks would also be desirable if we intend to control multiple sub-systems. DeepMind's Gato [54] and Robotics Transformer (RT-1) from Google Brain [55] have recently demonstrated the most promising strides towards artificial general intelligence, enabling multitask learning using a context-based generalized policy. Applicability of such frameworks that combine transformer models with reinforcement learning strategies indeed looks promising for current and future generation GW observatories, and our work is the first step in that direction.…”

Section: Discussionmentioning

confidence: 99%

First demonstration of neural sensing and control in a kilometer-scale gravitational wave observatory

Mukund¹,

Lough²,

Bisht³

et al. 2023

Preprint

View full text Add to dashboard Cite

Suspended optics in gravitational wave (GW) observatories are susceptible to alignment perturbations and, in particular, to slow drifts over time due to variations in temperature and seismic levels. Such misalignments affect the coupling of the incident laser beam into the optical cavities, degrade both circulating power and optomechanical photon squeezing, and thus decrease the astrophysical sensitivity to merging binaries. Traditional alignment techniques involve differential wavefront sensing using multiple quadrant photodiodes, but are often restricted in bandwidth and are limited by the sensing noise. We present the first-ever successful implementation of neural network-based sensing and control at a gravitational wave observatory and demonstrate low-frequency control of the signal recycling mirror at the GEO 600 detector. Alignment information for three critical optics is simultaneously extracted from the interferometric dark port camera images via a CNN-LSTM network architecture and is then used for MIMO control using soft actor-critic-based deep reinforcement learning. Overall sensitivity improvement achieved using our scheme demonstrates deep learning's capabilities as a viable tool for real-time sensing and control for current and next-generation GW interferometers.

show abstract

Section: Discussionmentioning

confidence: 99%

First demonstration of neural sensing and control in a kilometer-scale gravitational wave observatory

Mukund¹,

Lough²,

Bisht³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…With proper correction and prompting, the Transformer can generate valid actions in the embodied environment. Furthermore, similarly to Gato, RT-1 [Brohan et al, 2022] leverages largescale datasets with diverse robotics experiences and language instructions to train a Transformer as well as a tokenizer, which achieves high performance on downstream tasks.…”

Section: Generalize To Multiple Domainsmentioning

confidence: 99%

A Survey on Transformers in Reinforcement Learning

Li¹,

Luo²,

Lin³

et al. 2023

Preprint

View full text Add to dashboard Cite

Transformer has been considered the dominating neural architecture in NLP and CV, mostly under a supervised setting. Recently, a similar surge of using Transformers has appeared in the domain of reinforcement learning (RL), but it is faced with unique design choices and challenges brought by the nature of RL. However, the evolution of Transformers in RL has not yet been well unraveled. Hence, in this paper, we seek to systematically review motivations and progress on using Transformers in RL, provide a taxonomy on existing works, discuss each sub-field, and summarize future prospects.

show abstract

“…IL methods can be roughly divided into two types: behavior cloning (BC) and inverse reinforcement learning (IRL). Although behavior cloning may seem simple, it is widely used in practice, even for challenging tasks like real-world robot manipulation [14], [15]. IRL is a bit more complex because it first estimates a reward function from the demonstration data and then solves a forward RL problem using this reward function.…”

Section: A Learning Policymentioning

confidence: 99%

“…This can be particularly useful in complex or dynamic environments, where trial-anderror learning may be impractical or infeasible. For example, learning from demonstrations has been used to train agents to perform tasks such as robot manipulation [14], [15], where the expert knowledge of a human operator can be used to guide the learning process. Another advantage of leveraging demonstrations is that it can reduce the amount of data and computational resources required for training.…”

Section: Introductionmentioning

confidence: 99%

Boosting Reinforcement Learning and Planning with Demonstrations: A Survey

Mu¹,

Sheng²

2023

Preprint

View full text Add to dashboard Cite

Although reinforcement learning has seen tremendous success recently, this kind of trial-and-error learning can be impractical or inefficient in complex environments. The use of demonstrations, on the other hand, enables agents to benefit from expert knowledge rather than having to discover the best action to take through exploration. In this survey, we discuss the advantages of using demonstrations in sequential decision making, various ways to apply demonstrations in learning-based decision making paradigms (for example, reinforcement learning and planning in the learned models), and how to collect the demonstrations in various scenarios. Additionally, we exemplify a practical pipeline for generating and utilizing demonstrations in the recently proposed ManiSkill robot learning benchmark.

show abstract

RT-1: Robotics Transformer for Real-World Control at Scale

Cited by 37 publications

References 0 publications

First demonstration of neural sensing and control in a kilometer-scale gravitational wave observatory

First demonstration of neural sensing and control in a kilometer-scale gravitational wave observatory

A Survey on Transformers in Reinforcement Learning

Boosting Reinforcement Learning and Planning with Demonstrations: A Survey

Contact Info

Product

Resources

About