This work proposes a framework for the optimization of the metrological performances of a system that measures the relative position and orientation between two surfaces. The method is based on the creation of a non-linear measurement model of the instrument. The method uses the Monte Carlo method and the Design of Experiments techniques for determining the instrument uncertainty and the uncertainty sensitivity versus the geometrical and metrological instrument characteristics. The result of the proposed approach is a nonlinear simplified numerical model of the measurement uncertainty and of the bias error components, that is used for the instrument design and compensation. A case-study related to a misalignment measurement system based on a universal joint is presented. The measurement uncertainty computed with the proposed method has been compared with the one obtained in fit-to-purpose experiments performed with a robotic manipulator. An optimal calibration procedure is then used in order to identify the parameters of the system minimizing the overall uncertainty.