The optimization design of gear transmission based on MATLAB is studied for the purpose of finding a convenient and practical method. A mathematical model of optimization design is established in order to minimize the volume of gear transmission, the sequential quadratic programming method (SQP) is used to solve the nonlinear constrained optimization problems. The method how to solve gear parameter optimization base on optimization toolbox of MATLAB is introduced in detail. An optimization design example is given, the optimization result accords with the standard specifications of mechanical design and can be directly used. The result shows that this method is convenient and effective, and can simplify computer programming, is especially suitable for engineer.