The tt * equations define a flat connection on the moduli spaces of 2d, N = 2 quantum field theories. For conformal theories with c = 3d, which can be realized as nonlinear sigma models into Calabi-Yau d-folds, this flat connection is equivalent to special geometry for threefolds and to its analogs in other dimensions. We show that the non-holomorphic content of the tt * equations in the cases d = 1, 2, 3 is captured in terms of finitely many generators of special functions, which close under derivatives. The generators are understood as coordinates on a larger moduli space. This space parameterizes a freedom in choosing representatives of the chiral ring while preserving a constant topological metric. Geometrically, the freedom corresponds to a choice of forms on the target space respecting the Hodge filtration and having a constant pairing. Linear combinations of vector fields on that space are identified with generators of a Lie algebra. This Lie algebra replaces the non-holomorphic derivatives of tt * and provides these with a finer and algebraic meaning. For sigma models into lattice polarized K3 manifolds, the differential ring of special functions on the moduli space is constructed, extending known structures for d = 1 and 3. The generators of the differential rings of special functions are given by quasi-modular forms for d = 1 and their generalizations in d = 2, 3. Some explicit examples are worked out including the case of the mirror of the quartic in P 3 , where due to further algebraic constraints, the differential ring coincides with quasi modular forms.