In this paper we present a method for nonlinear frequency response analysis of mechanical vibrations of 3-dimensional solid structures. For computing nonlinear frequency response to periodic excitations, we employ the wellestablished harmonic balance method. A fundamental aspect for allowing a large-scale application of the method is model order reduction of the discretized equation of motion. Therefore we propose the utilization of a modal projection method enhanced with modal derivatives, providing second-order information. For an efficient spatial discretization of continuum mechanics nonlinear partial differential equations, including large deformations and hyperelastic material laws, we use the isogeometric finite element method, which has already been shown to possess advantages over classical finite element discretizations in terms of higher accuracy of numerical approximations in the fields of linear vibration and static large deformation analysis. With several computational examples, we demonstrate the applicability and accuracy of the modal derivative reduction method for nonlinear static computations and vibration analysis. Thus, the presented method opens a promising perspective on application of nonlinear frequency analysis to large-scale industrial problems.