Bruinier and Yang conjectured a formula for intersection numbers on an arithmetic Hilbert modular surface, and as a consequence obtained a conjectural formula for CM(K).G 1 under strong assumptions on the ramification in K. Yang later proved this conjecture under slightly stronger assumptions on the ramification. In recent work, Lauter and Viray proved a different formula for CM(K).G 1 for primitive quartic CM fields with a mild assumption, using a method of proof independent from that of Yang. In this paper we show that these two formulas agree, for a class of primitive quartic CM fields which is slightly larger than the intersection of the fields considered by Yang and Lauter and Viray. Furthermore, the proof that these formulas agree does not rely on the results of Yang or Lauter and Viray. As a consequence of our proof, we conclude that the Bruinier-Yang formula holds for a slightly largely class of quartic CM fields K than what was proved by Yang, since it agrees with the Lauter-Viray formula, which is proved in those cases. The factorization of these intersection numbers has applications to cryptography: precise formulas for them allow one to compute the denominators of Igusa class polynomials, which has important applications to the construction of genus 2 curves for use in cryptography.
The Bruinier-Yang formulaIn this section, we describe the formula for (CM(K).G 1 ) ℓ that was conjectured by Bruinier and Yang [BY06] and later proved by Yang [Yan]. One of the factors (which we denote by R K/ F ) in this intersection formula is multiplicative, so it makes sense to express it in terms of local factors over each prime. The key result of this section is Theorem 2.5, which does 3