In the top-down holographic model of QCD based on D4/D8-branes in type IIA string theory and some of the bottom-up models, the low energy effective theory of mesons is described by a five-dimensional Yang-Mills-Chern-Simons theory in a certain curved background with two boundaries. The fivedimensional Chern-Simons term plays a crucial role in reproducing the correct chiral anomaly in fourdimensional massless QCD. However, there are some subtle ambiguities in the definition of the Chern-Simons term for the cases with topologically nontrivial gauge bundles, which include the configurations with baryons. In particular, for the cases with three flavors, it was pointed out by Hata and Murata that the naive Chern-Simons term does not lead to an important constraint on the baryon spectrum, which is needed to pick out the correct baryon spectrum observed in nature. In this paper, we propose a formulation of a well-defined Chern-Simons term which can be used for the cases with baryons, and show that it recovers the correct baryon constraint as well as the chiral anomaly in QCD.