Abstract-Surgeries of the skull base require accuracy to safely navigate the critical anatomy. This is particularly the case for endoscopic endonasal skull base surgery (ESBS) where the surgeons work within millimeters of neurovascular structures at the skull base. Today's navigation systems provide approximately 2 mm accuracy. Accuracy is limited by the indirect relationship of the navigation system, the image and the patient. We propose a method to directly track the position of the endoscope using video data acquired from the endoscope camera. Our method first tracks image feature points in the video and reconstructs the image feature points to produce three-dimensional (3D) points, and then registers the reconstructed point cloud to a surface segmented from pre-operative computed tomography (CT) data. After the initial registration, the system tracks image features and maintains the two-dimensional (2D)-3D correspondence of image features and 3D locations. These data are then used to update the current camera pose. We present a method for validation of our system, which achieves sub-millimeter (0.70 mm mean) target registration error (TRE) results.