Simple cells in the visual cortex process spatial as well as temporal information of the visual stream and enable the perception of motion information. Previous work suggests that the direction selectivity of V1 simple cells is associated with a temporal offset in the thalamocortical input stream through lagged and non-lagged cells of the lateral geniculate nucleus (LGN), but also with intercortical inhibition. While there exists a large corpus of models for spatiotemporal receptive fields, the majority of them built-in the spatiotemporal dynamics by utilizing a combination of spatial and temporal functions and thus, do not explain the emergence of spatiotemporal dynamics on basis of network dynamics emerging in the retina and the LGN. In order to better comprehend the emergence of spatiotemporal processing and direction selectivity, we used a spiking neural network to implement the visual pathway from the retina to the primary visual cortex. By varying different functional parts in our network, we demonstrate how the direction selectivity of simple cells emerges through the interplay between two components: tuned intercortical inhibition and a temporal offset in the feedforward path through lagged LGN cells. Further, we observe that direction-selective simple cells are linked to a particular spiking pattern in a local excitatory-inhibitory circuit: If the stimulus moves in the non-preferred direction of a simple cell, inhibitory neurons with a different spatial position or tuning spike earlier, preventing the simple cell to spike. However, in the preferred direction, these inhibitory cells spike later, enabling the simple cell to spike.