In this work, the effect of nitrogen doping on vanadium micro-alloyed P460NL1 steel is studied in terms of microstructures and impact toughness. As the nitrogen content increased from 0.0036% to 0.0165%, the number of V (C,N) particles increased. The fine precipitates of V (C,N) effectively pin the prior austenite grain boundary, resulting in the refinement of the austenite grain. The intragranular and intergranular V-containing coarse particles enhanced the nucleation of intragranular ferrite and the grain boundaries of polygonal ferrite during air cooling. Accordingly, the proportion of heterogeneously nucleated ferrite increased, and the grain size of ferrite decreased. Notably, the size of the pearlite microstructure decreased, and the bainite microstructure appeared with a high doping of N. With the increase in N content, the impact toughness of vanadium micro-alloyed P460NL1 steel was enhanced. This can be attributed to the refinement of ferrite and the reduction in pearlite, which, in turn, was ascribed to the increase in nitrogen.