Large-Scale Optimization for Evaluation Functions with Minimax Search

Abstract

This paper presents a new method, Minimax Tree Optimization (MMTO), to learn a heuristic evaluation function of a practical alpha-beta search program. The evaluation function may be a linear or non-linear combination of weighted features, and the weights are the parameters to be optimized. To control the search results so that the move decisions agree with the game records of human experts, a well-modeled objective function to be minimized is designed. Moreover, a numerical iterative method is used to find local minima of the objective function, and more than forty million parameters are adjusted by using a small number of hyper parameters. This method was applied to shogi, a major variant of chess in which the evaluation function must handle a larger state space than in chess. Experimental results show that the large-scale optimization of the evaluation function improves the playing strength of shogi programs, and the new method performs significantly better than other methods. Implementation of the new method in our shogi program Bonanza made substantial contributions to the program's first-place finish in the 2013 World Computer Shogi Championship. Additionally, we present preliminary evidence of broader applicability of our method to other two-player games such as chess.

Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm

“…Kaneko and Hoki (16) trained the weights of a shogi evaluation function comprising a million features, by learning to select expert human moves during alpha-beta serach. They also performed a large-scale optimization based on minimax search regulated by expert game logs (12); this formed part of the Bonanza engine that won the 2013 World Computer Shogi Championship.…”

Reinforcement Learning to Create Value and Policy Functions Using Minimax Tree Search in Hex

“…Traditionally, evaluation functions have been created by combining manually quantified position and move features. Many methods have been proposed for creating evaluation functions using extracted features, and their effectiveness has been demonstrated [3], [4]. However, it is difficult to create highly accurate functions using such extracted features.…”