Over the past decades, many publications on the use of genetic algorithms, which offer a new and powerful approach for solving the problem of power system mode optimization, have appeared. Despite this, the issues of effectively taking into account various constraints when solving such problems with genetic algorithms remain opened. In this regard, this article proposes an algorithm for optimizing power system modes by genetic algorithm, taking into account functional constraints in the form of equalities and inequalities by various penalty functions. The results of effectiveness research of the given algorithm in the example of optimization of 8-nodal power system with four thermal power plants and three lines with controlled power flows are presented.