We describe a coaxial holographic recording system for achieving high recording density. We implement several techniques, such as an objective lens with high numerical aperture (NA), high capacity page data format, a random binary phase mask, and an optical noise reduction element. Our system successfully realizes a hologram recording/retrieving at a low diffraction efficiency less than 2.0 x 10(-3) and achieves a raw data density of 180 Gbit/in.(2), thus demonstrating the potential of a coaxial holographic system for high-density optical storage systems.