A generalization Gal ℓ (p, q) of the conformal Galilei algebra g ℓ (d) with Levi subalgebra isomorphic to sl(2, R) ⊕ so(p, q) is introduced and a virtual copy of the latter in the enveloping algebra of the extension is constructed. Explicit expressions for the Casimir operators are obtained from the determinant of polynomial matrices. For the central factor Gal ℓ (p, q), an exact formula giving the number of invariants is obtained and a procedure to compute invariants functions that do not depend on variables of the Levi subalgebra is developed. It is further shown that such solutions determine complete sets of invariants provided that the relation d ≤ 2ℓ + 2 is satisfied.