The delicacy of gauge choice in calculating atomic transitions was first raised by Lamb and gave arise to intensive discussion as well as much controversy. These discussion and controversy focused on choosing a proper gauge for the electromagnetic wave that interacts with an atom. The issue was claimed to have been solved, especially by Lamb himself and co-workers, by favoring a gaugeinvariant Hamiltonian for defining the atomic state in the presence of electromagnetic wave. Here we extend the problem to include a time-dependent relativistic non-perturbative Coulomb field, which can be produced by a cluster of relativistic charged particles. If adiabatic conditions are carefully maintained, such a field must be included along side the nuclear Coulomb potential when defining the atomic state. We reveal that when taking the external field approximation, the gauge choice for this time-dependent relativistic non-perturbative Coulomb field cannot be overcome by previous method, and leads to considerable gauge-dependence of the transient spontaneous radiation spectrum. We calculate explicitly with a simple one-dimensional charged harmonic oscillator that such a gauge-dependence can be of a measurable magnitude of 10 MHz or larger for the commonly used Coulomb, Lorentz, and multipolar gauges. Contrary to the popular view, we explain that this gauge dependence is not really a disaster, but actually an advantage here: The relativistic bound-state problem is so complicated that a fully quantum-field method is still lacking, thus the external field approximation cannot be derived and hence not guaranteed. However, by fitting to the experimental data, one may always define an effective external field, which may likely be parameterized with the gauge potential in a particular gauge. This effective external field would not only be of phenomenological use, but also shed light on the physical significance of the gauge field.