Vision-based system identification and 3D keypoint discovery using dynamics constraints

Abstract

This paper introduces V-SysId, a novel method that enables simultaneous keypoint discovery, 3D system identification, and extrinsic camera calibration from an unlabeled video taken from a static camera, using only the family of equations of motion of the object of interest as weak supervision. V-SysId takes keypoint trajectory proposals and alternates between maximum likelihood parameter estimation and extrinsic camera calibration, before applying a suitable selection criterion to identify the track of interest. This is then used to train a keypoint tracking model using supervised learning. Results on a range of settings (robotics, physics, physiology) highlight the utility of this approach.

KeyCLD: Learning constrained Lagrangian dynamics in keypoint coordinates from images

“…14, 15), unactuated, underactuated and fully actuated cartpole environment (Fig. 16,17,18) and unactuated, underactuated and fully actuated acrobot environment (Fig. 19,20,21).…”

“…Machine Learning and the Physical Sciences Workshop, NeurIPS 2023. human pose estimation [14], control and robotic manipulation [15,16], system identification and dynamic modelling [17]. Jakab et al [18] learn a keypoint representation unsupervised by using it as an information bottleneck for reconstructing images.…”