Identification and Adaptive Control of Markov Jump Systems: Sample Complexity and Regret Bounds

Abstract

Learning how to effectively control unknown dynamical systems is crucial for intelligent autonomous systems. This task becomes a significant challenge when the underlying dynamics are changing with time. Motivated by this challenge, this paper considers the problem of controlling an unknown Markov jump linear system (MJS) to optimize a quadratic objective. By taking a model-based perspective, we consider identification-based adaptive control for MJSs. We first provide a system identification algorithm for MJS to learn the dynamics in each mode as well as the Markov transition matrix, underlying the evolution of the mode switches, from a single trajectory of the system states, inputs, and modes. Through mixing-time arguments, sample complexity of this algorithm is shown to be O(1 √ T ). We then propose an adaptive control scheme that performs system identification together with certainty equivalent control to adapt the controllers in an episodic fashion. Combining our sample complexity results with recent perturbation results for certainty equivalent control, we prove that when the episode lengths are appropriately chosen, the proposed adaptive control scheme achieves O( √ T ) regret, which can be improved to O(polylog(T )) with partial knowledge of the system. Our proof strategy introduces innovations to handle Markovian jumps and a weaker notion of stability common in MJSs. Our analysis provides insights into system theoretic quantities that affect learning accuracy and control performance. Numerical simulations are presented to further reinforce these insights.

System Identification via Nuclear Norm Regularization

“…Refs. [46,47] study clustering and identification for Markov jump system and Ref. [48] further analyzes the optimal control strategy based on the estimated system parameters.…”

Finite-sample analysis of identification of switched linear systems with arbitrary or restricted switching

“…The statistical analysis of this identification problem has been considered in [9], [10], [11]. The authors of [9] consider a more general setup where the outputs are measured instead of the states, and a subspace identification method is employed.…”

“…However, the estimation procedure requires collecting data from multiple independent trajectories obtained by restarting the system to estimate the Hankel matrix. With measured states and measured switching signal, the statistical analysis of the switched LS estimator, i.e., applying the standard LS estimator for every mode separately, has been considered in [10], [11]. The extension of the analysis from the standard LS to switched LS is non-trivial, as the covariances of the local estimators are coupled through the hybrid system dynamics [6].…”

“…The authors in [11] have established the consistency of the switched LS estimator; however, the result is asymptotic and thus valid only when the data length approaches infinity. The work [10] addresses the finite-sample analysis of the switched LS estimator. The employed estimator requires subsampling the data and knowing the noise covariance, typically an unknown quantity.…”

Mode Reduction for Markov Jump Systems

“…Other than constructing and analyzing the reduced MJS, the technical tools we develop in this work regarding perturbations can be applied to cases when there are model mismatches, e.g. system estimation errors incurred when dynamics are learned in identification or data-driven adaptive control as in [11].…”