At present, with the rapid development of software, software security is very more and more serious. In order to solve the software security problem, it is necessary to monitor the software behavior. Therefore, this paper studies the differences between the running states of the software, and finds out and quantifies the differences, which further monitors the software behavior in software security. This paper analyzes the topology structure algorithm of software behavior model and proposes FSM Diff algorithm based on finite state automaton. FSM Diff algorithm is more effective to select the key state nodes. Besides, experiment shows that FSM Diff algorithm is effective to select the key state nodes in the software running state diagram.