We study special cycles on a Shimura variety of orthogonal type over a totally real field of degree d associated with a quadratic form in n`2 variables whose signature is pn, 2q at e real places and pn`2, 0q at the remaining d´e real places for 1 ď e ă d. Recently, these cycles were constructed by Kudla and Rosu-Yott and they proved that the generating series of special cycles in the cohomology group is a Hilbert-Siegel modular form of half integral weight. We prove that, assuming the Beilinson-Bloch conjecture on the injectivity of the higher Abel-Jacobi map, the generating series of special cycles of codimension er in the Chow group is a Hilbert-Siegel modular form of genus r and weight 1`n{2. Our result is a generalization of Kudla's modularity conjecture, solved by Yuan-Zhang-Zhang unconditionally when e " 1.