OSCAR: Data-Driven Operational Space Control for Adaptive and Robust Robot Manipulation

Abstract

Learning performant robot manipulation policies can be challenging due to high-dimensional continuous actions and complex physics-based dynamics. This can be alleviated through intelligent choice of action space. Operational Space Control (OSC) has been used as an effective taskspace controller for manipulation. Nonetheless, its strength depends on the underlying modeling fidelity, and is prone to failure when there are modeling errors. In this work, we propose OSC for Adaptation and Robustness (OSCAR), a data-driven variant of OSC that compensates for modeling errors by inferring relevant dynamics parameters from online trajectories. OSCAR decomposes dynamics learning into task-agnostic and task-specific phases, decoupling the dynamics dependencies of the robot and the extrinsics due to its environment. This structure enables robust zero-shot performance under out-of-distribution and rapid adaptation to significant domain shifts through additional finetuning. We evaluate our method on a variety of simulated manipulation problems, and find substantial improvements over an array of controller baselines. For more results and information, please visit https://cremebrule.github.io/oscar-web/.

Guided Reinforcement Learning: A Review and Evaluation for Efficient and Effective Real-World Robotics [Survey]

“…State representation [24], [53], [86], [87], [97] [22], [55], [138] Reward design [38], [119] [18] [93] [135] [23], [31], [33], [54] Abstract learning [27] [106], [107] [3], [16], [82], [134], [136] Offline RL [26] [1], [20], [39], [63], [116], [133], [140] Parallel learning [48], [114] [11], [32], [44], [58], [79], [80], [88], [113] Learning from demonstration [7], [35] [19]…”

“…For instance, Martin-Martin et al [82] introduce variable impedance control in the end-effector space to simplify exploration and improve robustness to disturbances. Wong et al [136] introduce Operational Space Control for Adaption and Robustness, a data-driven version of operational space control [59] that is adaptive to changes in the dynamics of a manipulation setting. Bogdanovic et al [16] propose a policy learning the impedance and desired position in the joint space and compare this approach to torque control and a fixed gain proportional-derivative controller.…”

“…Potential directions include bioinspired reward shaping [119], parameter optimization [100], and inverse RL [37] for automatically finding appropriate reward functions. Finally, selecting an abstracted action space instead of a complex one can largely simplify the training task (e.g., [106] and [136]). We see a potential challenge in choosing appropriate levels and methods of abstractions, such as the joint space and task space [82].…”

“…Carefully selecting taskspecific state representations, reward functions, and action spaces can improve both the time to convergence and performance [82], [86], [119]. RL approaches also can be combined with classical control to learn in state spaces of lower complexity [27], [136]. Finally, integrating knowledge about the learning task structure has been found to improve performance and accelerate convergence [96], [139].…”

Just Round: Quantized Observation Spaces Enable Memory Efficient Learning of Dynamic Locomotion

“…While modelbased learning has been shown to work well in some complex dynamic environments [26], model-free methods remain a popular choice in the dynamic locomotion community [16], [17]. Others have turned to embedded and more descriptive action spaces [27], [28], [29], [30] and reduced order models [23] to enable more robust and sample efficient learning. However, these efforts have mainly ignored the impact of observation space compression on model-free learning.…”