We present a numerical method to approximate the memory functions in the generalized Langevin models for the collective dynamics of macromolecules. We first derive the exact expressions of the memory functions, obtained from projection to subspaces that correspond to the selection of coarse-grain variables. In particular, the memory functions are expressed in the forms of matrix functions, which will then be approximated by Krylov-subspace methods. It will also be demonstrated that the random noise can be approximated under the same framework, and the second fluctuation-dissipation theorem is automatically satisfied. The accuracy of the method is examined through several numerical examples.