We present measurements of the rest-frame ultraviolet luminosity function (UV LF) at redshifts 𝑧 = 3, 𝑧 = 4 and 𝑧 = 5, using 96894, 38655 and 7571 sources respectively to map the transition between AGN and galaxy-dominated ultraviolet emission shortly after the epoch of reionization. Sources are selected using a comprehensive photometric redshift approach, using 10 deg 2 of deep extragalactic legacy fields covered by both HSC and VISTA. The use of template fitting spanning a wavelength range of 0.3-2.4𝜇m achieves 80-90 per cent completeness, much higher than classical colour-colour cut methodology. The measured LF encompasses −26 < 𝑀 UV < −19.25(−20.5) at 𝑧 = 3(5). This is further extended to −28.5 < 𝑀 UV < −16 using complementary results from other studies, allowing for the simultaneous fitting of the combined AGN and galaxy LF. We find that there are fewer UV luminous galaxies (𝑀 UV < −22) at 𝑧 ∼ 3 than 𝑧 ∼ 4, indicative of an onset of widespread quenching alongside dust obscuration, and that the evolution of the AGN LF is much more rapid than the galaxy LF, with their number density rising by around 2 orders of magnitude from 3 < 𝑧 < 6. We also find that it remains difficult to determine if a double power law (DPL) functional form is preferred over the Schechter function to describe the galaxy UV LF with photometric data alone. Estimating the Hydrogen ionizing photon budget from our UV LFs, we find that AGN can contribute to, but cannot solely maintain, the reionization of the Universe at 𝑧 = 3 − 5. However, the rapidly evolving AGN LF strongly disfavours a significant contribution within the EoR.