The eccentricity matrix E(G) of a connected graph G is obtained from the distance matrix of G by keeping the largest non-zero entries in each row and each column, and leaving zeros in the remaining ones. The eigenvalues of E(G) are the E-eigenvalues of G. In this article, we find the inertia of the eccentricity matrices of trees. Interestingly, any tree on more than 4 vertices with odd diameter has two positive and two negative E-eigenvalues (irrespective of the structure of the tree). A tree with even diameter has the same number of positive and negative E-eigenvalues, which is equal to the number of 'diametrically distinguished' vertices (see Definition 3.1). Besides we prove that the spectrum of the eccentricity matrix of a tree is symmetric with respect to the origin if and only if the tree has odd diameter. As an application, we characterize the trees with three distinct E-eigenvalues.