We study the relaxation of the center-of-mass, or dipole oscillations in the system of interacting fermions confined spatially. With the confinement frequency ω ⊥ fixed the particles were considered to freely move along one (quasi-1D) or two (quasi-2D) spatial dimensions. We have focused on the regime of rare collisions, such that the inelastic collision rate, 1/τin ≪ ω ⊥ . The dipole oscillations relaxation rate, 1/τ ⊥ is obtained at three different levels: by direct perturbation theory, solving the integral Bethe-Salpeter equation and applying the memory function formalism. As long as anharmonicity is weak, 1/τ ⊥ ≪ 1/τin the three methods are shown to give identical results. In quasi-2D case 1/τ ⊥ = 0 at zero temperature. In quasi-1D system 1/τ ⊥ ∝ T 3 if the Fermi energy, EF lies below the critical value, EF < 3ω ⊥ /4. Otherwise, unless the system is close to integrability, the rate 1/τ ⊥ has the temperature dependence similar to that in quasi-2D. In all cases the relaxation results from the excitation of particle-hole pairs propagating along unconfined directions resulting in the relationship 1/τ ⊥ ∝ 1/τin, with the inelastic rate 1/τin = 0 as the phase-space opens up at finite energy of excitation, ω ⊥ . While 1/τ ⊥ ∝ τin in the hydrodynamic regime, ω ⊥ ≪ 1/τin, in the regime of rare collisions, ω ⊥ ≫ 1/τin, we obtain the opposite trend 1/τ ⊥ ∝ 1/τin.