Magnetron sputter deposition of metal targets over liquids allows producing colloidal solutions of small metal nanoparticles (NPs) without any additional reducing or stabilizing reagents. Despite that this synthetic approach is known for almost 15 years, the detailed mechanism of NP formation is still unclear. Detailed investigations must be carried out to better understand the growth mechanism and, ultimately, control the properties of the NPs. Here, the combination of the gold (Au) target and castor oil, a highly available green solvent, was chosen as a model system to investigate how different experimental parameters affect the growth of NPs. The effect of deposition time, applied sputter power, working gas pressure, and type of sputter plasma (direct current magnetron sputtering (DC-MS) vs. high-power impulse magnetron sputtering (HiPIMS)) on properties of Au NPs has been studied by UV-vis spectroscopy and transmission electron microscopy (TEM), and further supported by quantum-chemistry calculations and mass-spectrometry analysis. The mechanism of the Au NP formation includes the production of primary NPs and their subsequent aggregative growth limited by diffusion in the viscous castor oil medium. Final Au NPs have a narrow size distribution and a medium diameter of 2.4–3.2 nm when produced in DC-MS mode. The NP size can be increased up to 5.2 ± 0.8 nm by depositing in HiPIMS mode which, therefore, mimics energy and time-consuming post synthesis annealing.