We introduce a fidelity-based measure, D QC (t), to quantify the differences in the dynamics of classical versus quantum walks over a graph. We provide universal, graph-independent, analytic expressions of this quantumclassical dynamical distance, showing that at short times D QC (t) is proportional to the coherence of the walker, i.e., a genuine quantum feature, whereas at long times it depends only on the size of the graph. At intermediate times, D QC (t) does depend on the graph topology through its algebraic connectivity. Our results show that the difference in the dynamical behavior of classical and quantum walks is entirely due to the emergence of quantum features at short times. In the long-time limit, quantumness and the different nature of the generators of the dynamics, e.g., the open-system nature of classical walks and the unitary nature of quantum walks, are instead contributing equally.