The main result of this paper is that, if Γ is a finite connected 4-valent vertex-and edge-transitive graph, then either Γ is part of a well-understood family of graphs, or every non-identity automorphism of Γ fixes at most 1/3 of the vertices. As a corollary, we get a similar result for 3-valent vertex-transitive graphs.