In order to achieve the maximum power point tracking (MPPT) of Photovoltaic (PV) systems in the presence of time-varying stochastic operation conditions and various uncertainties/disturbances, a passivity-based fractional-order sliding-mode control (PbFoSMC) scheme is proposed. The design can be classified into two steps, i.e., (a) Construct a storage function in terms of the tracking error of DC-link voltage, DC-link current and q-axis current for the PV system, upon which the actual characteristics of each term is thoroughly analyzed. Moreover, the beneficial terms are carefully retained to enhance the dynamical responses of the closed-loop system while the detrimental terms are fully removed to realize a global control consistency; (b) Based on the passivized system, a fractional-order sliding-mode control (FoSMC) is incorporated as an additional input, which can considerably improve the control performance with the aim of rapid uncertainties/disturbances rejection. Four case studies, including the solar irradiance change, temperature variation, power grid voltage drop, and PV inverter parameter uncertainties, are undertaken to evaluate the effectiveness of PbFoSMC in comparison to that of PID control, passivity-based control (PBC), and sliding-mode control (SMC), respectively. At last, a dSpace based hardware-in-loop (HIL) test is carried out to validate the implementation feasibility of PbFoSMC.