Memristive devices have demonstrated rich switching behaviors that closely resemble synaptic functions and provide a building block to construct efficient neuromorphic systems. It is demonstrated that resistive switching effects are controlled not only by the external field, but also by the dynamics of various internal state variables that facilitate the ionic processes. The internal temperature, for example, works as a second‐state variable to regulate the ion motion and provides the internal timing mechanism for the native implementation of timing‐ and rate‐based learning rules such as spike timing dependent plasticity (STDP). In this work, it is shown that the 2nd state‐variable in a Ta2O5‐based memristor, its internal temperature, can be systematically engineered by adjusting the material properties and device structure, leading to tunable STDP characteristics with different time constants. When combined with an artificial post‐synaptic neuron, the 2nd‐order memristor synapses can spontaneously capture the temporal correlation in the input streaming events.