We give a direct proof that the Mazur-Tate and Coleman-Gross heights on elliptic curves coincide. The main ingredient is to extend the Coleman-Gross height to the case of divisors with non-disjoint support and, doing some p-adic analysis, show that, in particular, its component above p gives, in the special case of an ordinary elliptic curve, the p-adic sigma function. We use this result to give a short proof of a theorem of Kim characterizing integral points on elliptic curves in some cases under weaker assumptions. As a further application, we give new formulas to compute double Coleman integrals from tangential basepoints.