A quantum collision model (CM), also known as repeated interactions model, can be built from the standard microscopic framework where a system S is coupled to a white-noise bosonic bath under the rotating wave approximation, which typically results in Markovian dynamics. Here, we discuss how to generalize the CM construction to the case of frequency-dependent system-bath coupling, which defines a class of colored-noise baths. This leads to an intrinsically non-Markovian CM, where each ancilla (bath subunit) collides repeatedly with S at different steps. We discuss the illustrative example of an atom in front a mirror in the regime of non-negligible retardation times.