In this paper, we study the performance of a next-generation cellular backhaul network that is based on a hybrid architecture using radio-over-fiber (RoF) and millimeter-wave (MMW) techniques. We develop a mathematic model and comprehensively analyze the performance of a MMW/RoF-based backhaul downlink under the impacts of various physical layer impairments originated from both optical fiber and wireless links. More specifically, the effects of nonlinear distortion, chromatic dispersion, fading, and many types of noises including shot noise, thermal noise, amplifier noise, and relative intensity noise are investigated. The numerical results show that the nonlinear distortion, fiber dispersion, and wireless fading are key factors that limit the system performance. Setting the modulation index properly helps minimize the effect of nonlinear distortion while implementing dispersion shifted optical fibers could be used to reduce the impact of dispersion and as a result, they can improve the bit-error rate. Moreover, it is also verified that, to mitigate the effect of multipath fading, remote radio heads should be located as near the remote antenna units as possible.