In this note we introduce the transcendental part t(X) of the motive of a cubic fourfold X and prove that it is isomorphic to the (twisted) transcendental part h tr 2 (F (X)) in a suitable Chow-Künneth decomposition for the motive of the Fano variety of lines F (X). Then we prove that t(X) is isomorphic to the Prym motive associated to the surface S l ⊂ F (X) of lines meeting a general line l. If X is a special cubic fourfold in the sense of Hodge theory, and F (X) ≃ S [2] , with S a K3, then we show that t(X) ≃ t 2 (S)(1), where t 2 (S) is the transcendental motive. Therefore the motive h(X) is finite dimensional if and only if S has a finite dimensional motive. If X is very general then t(X) cannot be isomorphic to the (twisted) transcendental motive of a surface. We relate the existence of an isomorphism t(X) ≃ t 2 (S)(1) to conjectures by Hassett and Kuznetsov on the rationality of a special cubic fourfold. Finally we consider the case of cubic fourfolds X admitting a fibration over P 2 , whose fibers are either quadrics or del Pezzo surfaces of degree 6, and prove the isomorphism t 2 (S)(1) ≃ t(X), with S a K3 surface.