Model based testing is an automated process in which executable test cases are derived from behavioral models of a system. Model checking is another verification approach in which all reachable states of a system are generated. In the literature, there are different approaches which suggest using model checkers for model based test case generation. Since all possible states and paths are generated by the model checker, selecting different paths in the state space as test cases based on some coverage criteria seems a promising solution. However, all these approaches suffer from two main challenges. The first challenge is the state space explosion problem which prevents the model checker to generate all the states. The second one is generating redundant test cases (paths). Recently, methods using meta-heuristic and evolutionary approaches have been proposed to cope with these two problems. Therefore, exploring a portion of the state space to detect the test objectives using an optimization approach can be a proper way to manage the state space explosion and generates the optimal test suites with the least redundancy. In this paper, an approach is proposed using Bayesian optimization algorithm (BOA) to generate test cases for service oriented systems specified through graph transformation. In the proposed approach, test suite is a set of paths on the state space that starting from an initial state and leading to the states in which all the test objectives is satisfied. In this research, we have implemented BOA with three different structures in GROOVE, an open source toolset for designing and model checking graph transformation. Experimental results show that our solution generates better results in terms of coverage, memory usage and speed in different case studies in comparison with the existing approaches.