We investigate the old problem of the fast relaxation of collisionless N-body systems which are collapsing or perturbed, emphasizing the importance of (non-collisional) discreteness effects. We integrate orbit ensembles in fixed potentials, estimating the entropy to analyze the time evolution of the distribution function. These estimates capture the correct physical behavior expected from the 2nd Law of Thermodynamics, without any spurious entropy production. For self-consistent (i.e. stationary) samples, the entropy is conserved, while for non-self-consistent samples, it increases within a few dynamical times, stabilizing at a maximum (even in integrable potentials). Our results make transparent that the main ingredient for this fast collisionless relaxation is the discreteness (finite N) of gravitational systems in any potential. Additionally, in non-integrable potentials, the presence of chaotic orbits accelerates the entropy production. Contrary to the traditional violent relaxation scenario, our results indicate that a time-dependent potential is not necessary for this relaxation. For the first time, in connection with the Nyquist-Shannon theorem we derive the typical timescale T /τ cr ≈ 0.1N 1/6 for this discreteness-driven relaxation, with slightly weaker N-dependencies for non-integrable potentials with substantial fractions of chaotic orbits. This timescale is much smaller than the collisional relaxation time even for small-N systems such as open clusters and represents an upper limit for the relaxation time of real N-body collisionless systems. Additionally, our results reinforce the conclusion of Beraldo e Silva et al. (2017) that the Vlasov equation does not provide an adequate kinetic description of the fast relaxation of collapsing collisionless N-body systems.