Free space optical (FSO) communication can support various unmanned aerial vehiclesâ (UAVs) applications that require large capacity data transmission. In order to perform FSO communication between two terminals, it is essential to employ a pointing, acquisition, and tracking (PAT) system with an efficient and optimal performance. We report on the development of a common optical-path-based FSO communication system, tailored for applications in UAVs. The proposed system is equipped with a quadrant photodiode (QPD)-based PAT system without an additional beacon beam subsystem. The presented approach reduced the structural complexity and improved the tracking efficiency for the same size, weight, and power (SWaP). To achieve a robust FSO link in a dynamic UAV environment, the observability and controllability were obtained based on the linearized control according to the incident beam size on the QPD, which was verified by optical simulation and experiments. As a result, the QPD-based PAT system for implementing FSO links demonstrated an up to 4.25 times faster tracking performance. Moreover, the FSO link experimentally confirmed the 1.25 Gbps full-duplex error-free communication at a 50 m distance.