With the complexity and high demands on quality assurance (QA) of photon beam radiation therapy, end-to-end (E2E) QA is necessary to validate the entire treatment workflow from pre-treatment imaging to beam delivery. A polymer gel dosimeter is a promising tool for three-dimensional (3D) dose distribution measurement. The purpose of this study is to design a fast “one delivery” polymethyl methacrylate (PMMA) phantom with a polymer gel dosimeter for the E2E QA test of the photon beam. The one delivery phantom is composed of ten calibration cuvettes for the calibration curve measurement, two 10 cm gel dosimeter inserts for the dose distribution measurement, and three 5.5 cm gel dosimeters for the square field measurement. The one delivery phantom holder is comparable in size and shape to that of a human thorax and abdomen. In addition, an anthropomorphic head phantom was employed to measure the patient-specific dose distribution of a VMAT plan. The E2E dosimetry was verified by undertaking the whole RT procedure (immobilization, CT simulation, treatment planning, phantom set-up, imaged-guided registration, and beam delivery). The calibration curve, field size, and patient-specific dose were measured with a polymer gel dosimeter. The positioning error can be mitigated with the one-delivery PMMA phantom holder. The delivered dose measured with a polymer gel dosimeter was compared with the planned dose. The gamma passing rate is 86.64% with the MAGAT-f gel dosimeter. The results ascertain the feasibility of the one delivery phantom with a polymer gel dosimeter for a photon beam in E2E QA. The QA time can be reduced with the designed one delivery phantom.