The collective dynamics and structure of animal groups has attracted the attention of scientists from different disciplines. A variety of agent-based models has been proposed to account for the emergence of coordinated collective behavior from simple interaction rules. A common, simplifying assumption of such collective movement models, is the consideration of individual agents moving with a constant speed. In this work we critically re-asses this assumption underlying a vast majority of collective movement models. First, we discuss experimental data showcasing the omnipresent speed variability observed in different species of live fish and artificial agents (RoboFish). Based on theoretical considerations accounting for inertia and rotational friction, we derive a functional dependence of the turning response of individuals on their instantaneous speed, which is confirmed by experimental data. We investigate how the interplay of variable speed and speed-dependent turning affects self-organized collective behavior by implementing an agent-based model which accounts for both effects. We show, that besides average speed, the individual speed variability may have a dramatic impact on the emergent collective dynamics, as two groups differing only in their speed variability, and being otherwise identical in all other behavioral parameters, can be in two fundamentally different stationary states (polarized versus disordered). We find that the local coupling between group polarization and individual speed is strongest at the order-disorder transition, and that, in contrast to fixed speed models, the group's spatial extent does not have a maximum at the transition. Furthermore, we demonstrate a decrease in polarization with group size for groups of individuals with variable speed, and a sudden decrease in mean individual speed at a critical group size (N=4 for Voronoi interactions) linked to a topological transition from an all-to-all to a distributed spatial interaction network. Overall,