Ultrasonic additive manufacturing (UAM) has garnered significant interest in the aerospace and automotive industries for its structural lightweighting and multi-material joining capabilities. This paper details the investigation on the effect of process variables on the resultant microstructure of the built-up part using UAM for aluminum 6061. The degree of recrystallization is quantified, and an energy metric, defined using the Read–Shockley relationship, is used to build an energy map of the welded part. The total energy stored in the resultant weld interface microstructure is quantified as a fraction of the input and is found to be about 0.1%. The width, average grain size, and percentage of High Angle Grain Boundaries (% HAGB) were used to compare microstructures of builds prepared using different processing conditions. Welding subsequent weld layers was not found to affect the previous welded layers. The effect of vibration amplitude and travel speed on the as-built microstructure were investigated, and the width of the interface was found to more than double when the weld amplitude is increased from the threshold value for joining (23 μm) and then stabilize at higher weld amplitudes. A better understanding of the effect of processing parameters on as-welded microstructures will assist parameter selection for UAM.