Let G be any group for which there is a least j such that Z j = Z j+1 in the upper central series. Define the group of j-central automorphisms as the kernel of the natural homomorphism from Aut(G) to Aut (G/Z j). We offer sufficient conditions for IA(G) to have a useful direct product structure, and apply our results to certain finitely generated center-by-metabelian groups.