We describe a second-order accurate approach to sparsifying the off-diagonal matrix blocks in the hierarchical approximate factorization methods for solving sparse linear systems. These methods repeatedly sparsify the fill-in matrix blocks that arise in block Gaussian elimination, to compute an approximate factorization of the given matrix, assuming that the fill-in blocks are low-rank.