In this article, we investigate the dependence of nuclear temperature on the emitting source neutron-proton ($N/Z$) asymmetry with the light charged particles (LCPs) and the intermediate mass fragments (IMFs) generated from the intermediate-velocity sources in thirteen reaction systems with different $N/Z$ asymmetries, $^{64}\textrm{Zn}$ on $^{112}\textrm{Sn}$, and $^{70}\textrm{Zn}$, $^{64}\textrm{Ni}$ on $^{112,124}\textrm{Sn}$, $^{58,64}\textrm{Ni}$, $^{197}\textrm{Au}$, $^{232}\textrm{Th}$ at 40 MeV/nucleon. The apparent temperature values of LCPs and IMFs from different systems are deduced from the measured yields using two helium-related and eight carbon-related double isotope ratio thermometers, respectively. Then, the sequential decay effect on the experimental apparent temperature deduction with the double isotope ratio thermometers are quantitatively corrected explicitly with the aid of the quantum statistical model. The present treatment is an improvement compared to our previous works in which an indirect method was adopted to qualitatively take into account the sequential decay effect. A negligible $N/Z$ asymmetry dependence of the real temperature after the corrections is quantitatively addressed in the heavy-ion reactions at the present intermediate energy, that a change of 0.1 unit in source $N/Z$ asymmetry is corresponding to an absolute change in temperature on an order of 0.03 to 0.29 MeV in average for LCPs and IMFs. This conclusion is in close agreement with that inferred qualitatively via the indirect method in our previous works.