Recovering Markov models from closed-loop data

Abstract

Situations in which recommender systems are used to augument decision making are becoming prevalent in many application domains. Almost always, these prediction tools (recommenders) are created with a view to affecting behavioural change. Clearly, successful applications actuating behavioural change, affect the original model underpinning the predictor, leading to an inconsistency. This feedback loop is often not considered in standard so-called Big Data learning techniques which rely upon machine learning/statistical learning machinery. The objective of this paper is to develop tools that recover unbiased user models in the presence of recommenders. More specifically, we assume that we observe a time series which is a trajectory of a Markov chain R modulated by another Markov chain S, i.e. the transition matrix of R is unknown and depends on the current state of S. The transition matrix of the latter is also unknown. In other words, at each time instant, S selects a transition matrix for R within a given set which consists of known and unknown matrices. The state of S, in turn, depends on the current state of R thus introducing a feedback loop. We propose an Expectation-Maximization (EM) type algorithm, which estimates the transition matrices of S and R. Experimental results are given to demonstrate the efficacy of the approach.

Spatial Positioning Token (SPToken) for Smart Mobility

“…As a use case of the proposed DLT, we successfuly presented a blockchain-supported distributed RL algorithm to determine an unknown distribution of traffic patterns in a city. 10 Fig. 7: Experiment 3: Performance of two test cars, one using recommendations from the UBEV-based routing system, and the other one always using shortest path policy.…”

“…recommenders), or even due to probing of the environment as a part of the model building, affect the environment and consequently the model building itself. Recently a number of papers have appeared highlighting the problem of recommender design in closed loop [17,31,6,3,10,25]. Even in cases when there is a separation between the effect of a recommender and its environment, the problem of recommender design is complex in many real world settings due to the challenge of sampling and obtaining real time data at low cost.…”

“…In terms of ML, we borrow heavily from RL and Markov Decison Processes (MDPs), and in particular, crowdsourced ML. The literature on MDPs and RL algorithms is vast and we simply point the reader to the recent publications [10,11,16,30], in which some of this work is discussed. With specific regard to RL and mobility, some applications are presented in [2,13,21,24,33].…”

Bayesian classifier for Route prediction with Markov chains

“…, r L | C k ) = L t=1 P (r t | C k ). We intend to test other stochastic process models, such as the recently developed closed-loop Markov modulated Markov chains [12] in the near future.…”

Distributed Ledger Enabled Control of Tyre Induced Particulate Matter in Smart Cities

“…This work has given rise to not only new systems of engagement for citizens and their cities, but also to new fundamental research questions in a number of other disciplines. The study of machine learning over "closed loop" data sets is one such example (Epperlein et al, 2019). At a very high level, Smart City Research is about making best use of existing resources in our cities, as we try to manage congestion, pollution, food production, and maintain living standards in the face of ever increasing pressure on natural resources.…”