This paper explores the secrecy performance of the recently proposed intensity modulation/direct detection (IM/DD) terrestrial free-space quantum key distribution (QKD) system, by using a Gaussian-beam propagation model and considering the combined effects of atmospheric turbulence and legitimate transceivers' misalignment. Secrecy performance metrics including quantum bit error rate (QBER) and ergodic secret-key rate are newly derived in closedform expressions, taking into account all combined effects of turbulence-and misalignment-induced fading channels, the eavesdropper's location relative to the legitimate receiver, and receiver noises. To satisfy security constraints, the system designs based on the intensity modulation depth and beam waist of the Gaussian beam at the transmitter, and dualthreshold (D-T) selection at the receiver, are comprehensively discussed under turbulence and misalignment conditions as well as different eavesdropper's locations. Monte-Carlo (M-C) simulations are also implemented to verify the analytical results. Remarkably, this paper also offers the first framework in the literature to evaluate the secrecy performance of free-space optical (FSO) systems considering the eavesdropper's location under the effect of misalignment between legitimate transceivers.