Before We Can Find a Model, We Must Forget about Perfection

Abstract

With Reinforcement Learning we assume that a model of the world does exist. We assume furthermore that the model in question is perfect (i.e. it describes the world completely and unambiguously). This article will demonstrate that it does not make sense to search for the perfect model because this model is too complicated and practically impossible to find. We will show that we should abandon the pursuit of perfection and pursue Event-Driven (ED) models instead. These models are generalization of Markov Decision Process (MDP) models. This generalization is essential because nothing can be found without it. Rather than a single MDP, we will aim to find a raft of neat simple ED models each one describing a simple dependency or property. In other words, we will replace the search for a singular and complex perfect model with a search for a large number of simple models.

Language for Description of Worlds. Part 1: Theoretical Foundation

“…All we need is a sufficiently good model which works for us. In [8] we explained why trying to find a perfect model of the world is an overly ambitious goal. Nevertheless, we will assume that a perfect model of the world does exist and that model provides the definition of the world.…”

Language for Description of Worlds. Part 2: The Sample World

“…The agent can influence even those events which are not driven by her actions. This is described in [9]. For these events the agent may have some "preference" and by her "preferences" the agent could have some influence on whether an event will or will not occur.…”

“…For most authors, non-deterministic implies that for each possible event there is one precisely defined probability. In [8,9] we showed that the latter statement is too deterministic. Telling the exact probability of occurrence for each and every event would be an exaggerated requirement.…”