Task-irrelevant stimulus numbers can automatically modulate concurrent temporal tasks——leading to the phenomenon of number-time association (NTA). Recent research provides converging evidence that the NTA occurs at the stage of temporal memory. Specifically, a reference memory containing encoded duration information can be modified by perceptual/concurrent digits, i.e., a perceptual/concurrent digit-induced NTA. Here, with five experiments, we investigated whether another working memory (WM)-related mechanism was involved in the generation of NTAs and how this memory-induced NTA was related with the perception-induced NTA. We first explored whether similar NTA effects existed for mnemonic digits which disappeared before time encoding but were actively maintained in WM, i.e., a mnemonic digit-induced NTA. Experiments 1–3 demonstrated both types of NTAs. Further, we revealed a close relationship between the two types of NTAs in two contexts. First, the mnemonic digit-induced NTA also relied on a perceptual number-time co-occurrence at time encoding. We found that the mnemonic digits influenced subsequent temporal processing when a task-irrelevant constant number ‘5’ was presented during target encoding, but not when a non-numerical symbol was presented, suggesting that temporal representations in the reference memory could be accessed and modified by both sensory and postsensory numerical magnitudes through this number-time co-occurrence. Second, the effects of perceptual and mnemonic digits on temporal reproduction could cancel each other out. A congruency effect for perceptual and mnemonic digits (relying on memorization requirement) was demonstrated in Experiments 4 and 5. Specifically, a typical NTA was observed when the magnitudes of memorized and the perceptual/concurrent digits were congruent (both were large or small numbers), but not when they were incongruent (one small and one large numbers). Taken together, our study sheds new light on the mechanism of NTA.