The ability to describe a neural network depends greatly on the number of neurons. In this paper, we consider a method of obtaining an optimal neural network through a forgetting operation. It is shown that by using the proposed method, an optimum neural network can be constructed not only from neurons having large singular values of the coefficient matrix, but also from neurons having small singular values of the coefficient matrix.