A rotor airfoil design optimization framework based on a surrogate model and unsteady flow is constructed. Two optimization models of unsteady design are founded. A study of these models and comparison of design results based on steady and unsteady flows are conducted using the aforementioned optimization framework. The first optimization model aims to simultaneously reduce the time-averaged drag and time-averaged pitching moment. The Pareto front embodies the relation of drag and pitching moment of the rotor airfoil in an unsteady flow, which shows that a slight relaxation of the drag restrictions may yield a remarkable decrease in pitching moment, thereby improving the synthetic aerodynamic performance of the rotor airfoil. Optimization results also show that an unsteady optimization model must give consideration to low speed performance, which has been neglected by other researchers. Therefore, the second optimization model is designed to improve the time-averaged lift-to-drag ratio and the maximum lift coefficient at low speed. An additional case was designed to validate that a well-designed airfoil in a steady flow may be unsatisfactory in an unsteady flow. The results from the last case show that the unsteady pitching moment characteristic is not always proportional to the static characteristic at high speed.