The paper presents a new method for action scene detection in the martial arts movies based on audio feature. First audio information is divided into seven categories: silence, harmonic environmental sound, pure music, pure voice, voice with background music, environmental sound with background music and non-harmonic environmental sound. Then the non-harmonic environmental sound is divided into the sword sound, the stick sound, the unarmed fight sound, the object broken sound and other non-harmonic environmental sound using support vector machines (SVM). At last the method of detecting action scene according to audio classification is elaborated. Experimental results prove that the method is effective.