A key concept in reinforcement learning (RL) is that of a state space. A state space is an abstract representation of the world using which the statistical relations in the world can be described. The simplest form of RL, model free RL, is widely applied to explain animal behavior in numerous neuroscientific studies. More complex RL versions assume that animals build and store an explicit model of the world in memory. Inherent to the application of both these classes of RL to animal behavior are assumptions about the underlying state space formed by animals, especially in relation to their representation of time. Here, we explicitly list these assumptions and show that they have several problematic implications. We propose a solution for these problems by using a continuous time Markov renewal process model of the state space. We hope that our explicit treatment results in a serious consideration of these issues when applying RL models to real animals.