Prime power fold cyclic branched covers along smoothly slice knots all bound rational homology balls. This phenomenon, however, does not characterize slice knots. In this paper, we give a new construction of non-slice knots that have the above property. The sliceness obstruction comes from computing twisted Alexander polynomials, and we introduce new techniques to simplify their calculation.