We study fidelity decay by a uniform semiclassical approach, in the three perturbation regimes, namely, the perturbative regime, the Fermi-golden-rule (FGR) regime, and the Lyapunov regime. A semiclassical expression is derived for fidelity of initial Gaussian wave packets with width of the order √h (h being the effective Planck constant). Short time decay of fidelity of initial Gaussian wave packets is also studied, with respect to two time scales introduced in the semiclassical approach. In the perturbative regime, it is confirmed numerically that fidelity has the FGR decay before the Gaussian decay sets in. An explanation is suggested to a non-FGR decay in the FGR regime, which has been observed in a system with weak chaos in the classical limit, by using the Levy distribution as an approximation for the distribution of action difference. In the Lyapunov regime, it is shown that the average of the logarithm of fidelity may have roughly the Lyapunov decay within some time interval, in systems possessing large fluctuation of the finite-time Lyapunov exponent in the classical limit.