In this work, we study and analyze the performance of Raman + Erbium-Ytterbium codoped fiber hybrid optical amplifier (HOA) for an ultradense wavelength division multiplexing (UD-WDM) system having 100 channels. The system has been investigated considering initial values of channel spacing and data rate of 0.1 nm (12.5 GHz) and 100 GB/s, respectively. Initially, the two important WDM system parameters—wavelength and channel spacing—have been selected and then optimization of the proposed HOA has been performed in terms of EYDFA length, pump power and Er+ concentration to achieve higher values of average gain, Q-factor and lower gain variation ratio. The optimized configuration of the HOA results in the achievement of higher value of average gain, Q-factor and gain variation ratio of 47 dB, 14 and 0.14, respectively, which confirms its viability for UD-WDM system applications.