We present a second quantization description of frequency-based continuous variables quantum computation in the subspace of single photons. For this, we define frequency and time operators using the free field Hamiltonian and its Fourier transform, and show that these observables, when restricted to the one photon per mode subspace, reproduce the canonical position-momentum commutation relations. As a consequence, frequency and time operators can be used to define a universal set of gates in this particular subspace. We discuss the physical implementation of these gates as well as their effect on single photon states, and show that frequency and time variables can also be used to implement continuous variables quantum information protocols, in the same way than polarization is currently used as a two-dimensional quantum variable.