Usual paradigm in the theory of electron transport is related to the fact that the dielectric permittivity of the insulator is assumed to be constant, no time dispersion. We take into account the "slow" polarization dynamics of the dielectric layers in the tunnel barriers in the fluctuating electric fields induced by single-electron tunneling events and study transport in the single electron transistor (SET). Here "slow" dielectric implies slow compared to the characteristic time scales of the SET charging-discharging effects. We show that for strong enough polarizability, such that the induced charge on the island is comparable with the elementary charge, the transport properties of the SET substantially deviate from the known results of transport theory of SET. In particular, the Coulomb blockade is more pronounced at finite temperature, the conductance peaks change their shape and the current-voltage characteristics show the memory-effect (hysteresis). However, in contrast to SETs with ferroelectric tunnel junctions, here the periodicity of the conductance in the gate voltage is not broken, instead the period strongly depends on the polarizability of the gate-dielectric. We uncover the fine structure of the hysteresis-effect where the "large" hysteresis loop may include a number of "smaller" loops. Also we predict the memory effect in the current-voltage characteristics I(V ), with I(V ) = −I(−V ).