As one of the most successful approaches for recommendation, matrix factorization based Collaborative Filtering (CF) technique has received considerable attentions over the past years. In this paper, we propose an orthogonal matrix factorization model with graph regularization to preserve the consistency of the local structure both in user and item spaces, respectively. Instead of traditional alternating optimization method, a greedy sequential one is introduced to optimize a pair of coupled factor vector and its corresponding loading vector simultaneously each time, thus the original optimization problem is converted into the well-studied Multivariate Eigen Problem (MEP). Furthermore, multiple pairs of coupled eigen-vectors can be obtained in sequence. To guarantee nonrecurring of repetition of solutions, a novel dualdeflation technique is developed to incorporate into the sequential optimization. Experimental results on MovieLens and Each Movie data sets demonstrate that the proposed method is much more competitive compared with the state of the art matrix factorization based collaborative filtering methods.