We discuss the interaction of an open heavy meson ( D and D * for charm or B and B * for bottom) and a nucleon (N ) by considering the π, σ, ρ, and ω exchange potentials. We construct a potential model by respecting chiral symmetry for light quarks and spin symmetry for heavy quarks. Model parameters are adjusted by referring the phenomenological nuclear (CD-Bonn) potentials reproducing the low-energy N N scatterings. We show that the resulting interaction may accommodate DN and BN bound states with quantum number I(J P ) = 0(1/2 − ), and a BN bound state with I(J P ) = 1(1/2 − ). We find that, in the present potential model, the σ exchange potential plays an important role.