Conventional image-reject mixing based on Hartley and Weaver architectures cannot deal with the mixing spurs generated by the nonlinearity of the mixer, leading to a fundamental restriction on the instantaneous bandwidth and frequency tuning range. This paper proposes and demonstrates a new architecture for broadband image-reject mixing enabled uniquely by photonics. The RF signal and the electrical LO signal are converted into the optical domain and are then launched into the signal and LO ports of a 90-degree optical hybrid, respectively. The optical hybrid coupler introduces 0, π, π/2 and 3π/2 phase shifts to the input signals, so a new dimension for additional balanced detection is enabled, which can dramatically remove the undesirable mixing spurs and the common-mode noises. As a result, the proposed architecture suppresses the downconverted image and the nonlinear mixing spurs simultaneously, enabling a truly wideband microwave frequency mixer. A theoretical and experimental investigation is performed. More than 60-dB image rejection and mixing spur suppression is achieved over a 40-GHz working frequency range.