Construction site layouts need to be planned for safety and efficiency before the site operations can begin. However, current planning methods rely heavily on the experience of planners. Inexperienced planners can easily neglect objects or misestimate object size and location during planning. This research aims to develop a video-enabled dynamic site planner that enables planners to model on-site objects from images from closed-circuit television (CCTV) and helps planners to simulate the site in real-time virtual construction. Two main methods are developed in this site planner. The first is the rapid image planar modeling method, which enables users to rapidly construct 3-D geometric models of the on-site objects from the video image. The second is the video-based virtual construction planning method, which provides a process for site layout planning in a simulated virtual construction environment. We have implemented this site planner by using a game engine. After an example case pertaining to layout planning, the results show that this system can help planners identify potential problems that will occur in the actual site layout process. The system can also serve as an information platform to reduce the gap between construction simulation and reality.